LiveStre4m: Feed-Forward Live Streaming of Novel Views from Unposed Multi-View Video

arXiv:2604.06740 · cs.CV · Submitted 2026-04-08 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "LiveStre4m: Feed-Forward Live Streaming of Novel Views from Unposed Multi-View Video".

Title and authors: Tom: So, to recap, LiveStre4m is proposing a system that takes video from multiple angles where you don't know exactly where the camera was positioned—that’s unposed video—and it generates a novel viewpoint for you in real time without needing any prior calibration data.

Jane: That’s the big takeaway, and what I find really interesting is how they handle the hard part of knowing *where* the camera was even in the first place. They're skipping that whole slow process of figuring out camera positions from scratch and just predicting them directly from those input images using a vision transformer.

Lu: From a representation learning standpoint, that prediction is key because it sets up this whole pipeline for reconstructing three dee geometry using a spatial module. The idea is they build this three dee Gaussian splatting representation of the scene based on those predicted views.

Meng: But what does that mean for deployment? If we’re talking about shipping this to an app, how much memory are we talking about holding all those intermediate representations while it’s streaming?

Tom: The key is that they keep the processing time per frame really low. They report a reconstruction time of zero point zero seven seconds per frame at one thousand twenty-four by seven hundred sixty-eight resolution, which is incredibly fast compared to optimization based methods.

Jane: That speed is what makes it relevant for live streaming; it’s not just a slow computer rendering a view, it’s actually doing the work fast enough to keep up with video. And they ensure that the temporal consistency—meaning the frames flow smoothly—by using a neural interpolation network that employs diffusion transformers and then feeds that into a super resolution module.

Lu: The interpolation part is smart because it lets them create high-quality intermediate frames between two keyframes, which then get boosted by the super resolution to give you those final high-resolution novel view video streams.

Meng: So, the trade-off they found was interesting; if you want a really smooth frame rate above ten frames per second, they have to cut the resolution down to one thousand one hundred fifty-two by seven hundred sixty-eight. That’s a practical limitation for production systems.

Tom: Yeah, and the ablation studies showed that the spatial module is actually taking up most of their time, at fifty two point one milliseconds of processing time, which tells us where we need to focus our hardware optimization if we want to push this further.

Jane: So they’ve essentially created a self-contained system where pose prediction leads into geometry reconstruction and then temporal smoothing all in one feed forward pipeline. It addresses the lack of reliable calibration data head-on.

Lu: And they're aiming for something even more robust in the future, suggesting online pose refinement and lightweight bundle adjustment to make sure that geometric consistency stays perfect over a long video sequence.

Tom: It seems like a solid step forward for making novel view synthesis practical for live applications, but we still need to see how it handles really complex, dynamic scenes where the camera movement is unpredictable.

The paper's summary: Tom: So, LiveStre4m isn't just finished; they actually suggest some ways to make it better for real use by focusing on improving geometric consistency and visual fidelity in the next version of the model.

Jane: That makes sense because even a fast system can get shaky if the underlying three dee geometry isn't perfectly consistent across different video frames, so adding online pose refinement is a smart move.

Lu: Exactly, they want to incorporate online pose refinement and lightweight bundle adjustment. This means the model can constantly tweak its understanding of where the camera is without needing a slow re-calibration step every time.

Meng: From my production side, if we can get that geometric consistency improved, it means fewer frames will look jumpy or inconsistent when streaming over a long period, which is exactly what we need for reliable outputs.

Tom: And they also mentioned scaling up the training data for multi-view video, which they believe would really boost the generalization and the overall output quality of the system.

Jane: That sounds like a necessary step because training on more diverse video inputs should help it handle more different kinds of scenes better than it currently can.

Lu: The authors also touched on making the system even lighter; they’re looking at methods to keep things efficient while improving accuracy, which is a constant balancing act in this whole field.

Meng: Efficiency is everything for us; if you make it accurate but slow down the runtime, it just won't get deployed on consumer hardware. So, any improvements that keep the runtime low while boosting quality are what we’re really interested in seeing.

Tom: It seems like they’re pushing toward a system that is not only fast and pose free but also highly accurate and ready for more unpredictable real-world video streams.

Jane: And it really highlights that while feed forward models are great for speed, the next big hurdle is making them as geometrically robust as methods that rely on known camera parameters.

Lu: So the path forward involves combining their fast feed forward reconstruction with more traditional optimization techniques to nail down those underlying spatial relationships.

The paper's improvements: Tom: So we’re wrapping up on LiveStre4m: Feed-Forward Live Streaming of Novel Views from Unposed Multi-View Video, and basically, they’ve shown us a way to generate novel viewpoints in real time without needing any prior camera calibration data.

Jane: That’s the big picture: it tackles the problem of needing those slow structure from motion pipelines just to get started with novel view synthesis.

Lu: The implication here is that we can move away from needing perfect ground truth camera parameters for every input, which is a huge win for practical applications where you don't have access to that kind of data.

Meng: For deployment, this means we can build systems that are much faster at generating novel views than those heavy optimization based methods, which is pretty crucial when you’re dealing with live video.

Tom: They hit a reconstruction time of zero point zero seven seconds per frame at one thousand twenty-four by seven hundred sixty-eight resolution, which is incredibly fast compared to those older methods they mentioned.

Jane: And they also showed that they can achieve rendering quality on the Neuralthree deeVideo dataset that’s competitive with some state of the art methods, hitting a PSNR of twenty-one point one one at that resolution <ref:2604.06740#pg2>.

Lu: It really validates the combination of their spatial module for scene geometry and their neural interpolation network for temporal consistency, showing it works as a cohesive feed forward system.

Meng: But we have to remember those caveats; they noted that if you want frame rates above ten frames per second, you have to reduce the image resolution down to one thousand one hundred fifty-two by seven hundred sixty-eight.

Tom: That’s the trade off; quality versus streamability, and it shows they are thinking about real deployment constraints right from the start.

Jane: And they pointed out that while it's fast and pose free, you still have to be careful about how far away your novel view is from where you started, because image quality drops as the target viewpoint moves further away.

Lu: The authors also mentioned their future work, focusing on online pose refinement and lightweight bundle adjustment to make that geometric consistency even stronger over time.

Tom: So the future direction is definitely about making it more robust and accurate for longer video sequences by adding those refinement tools they mentioned.

Jane: This paper, LiveStre4m: Feed-Forward Live Streaming of Novel Views from Unposed Multi-View Video, gives us a really tangible tool for getting photorealistic novel viewpoints quickly in a streaming environment.

Lu: It’s a strong foundation because it proves that feed forward reconstruction can handle this task efficiently without the heavy computational overhead of traditional methods.

Meng: For me, it means we can start prototyping these features in real-time applications much sooner, rather than waiting for massive optimization pipelines to finish running.

Lalam: I’m confident that this approach is solid; the combination of pose prediction and spatial geometry reconstruction provides a very direct path to novel views when you only have unposed inputs.

Conclusion: Tom: So we’re wrapping up on LiveStre4m: Feed-Forward Live Streaming of Novel Views from Unposed Multi-View Video, and essentially, they’ve shown us a way to generate novel viewpoints in real time without needing any prior camera calibration data.

Jane: That’s the big picture: it tackles the problem of needing those slow structure from motion pipelines just to get started with novel view synthesis.

Lu: The implication here is that we can move away from needing perfect ground truth camera parameters for every input, which is a huge win for practical applications where you don't have access to that kind of data.

Meng: For deployment, this means we can build systems that are much faster at generating novel views than those heavy optimization based methods, which is pretty crucial when you’re dealing with live video.

Tom: They hit a reconstruction time of zero point zero seven seconds per frame at one thousand twenty-four by seven hundred sixty-eight resolution, which is incredibly fast compared to those older methods they mentioned.

Jane: And they also showed that they can achieve rendering quality on the Neuralthree deeVideo dataset that’s competitive with some state of the art methods, hitting a PSNR of twenty one point one at that resolution.

Lu: It really validates the combination of their spatial module for scene geometry and their neural interpolation network for temporal consistency, showing it works as a cohesive feed forward system.

Meng: But we have to remember those caveats; they noted that if you want frame rates above ten frames per second, you have to reduce the image resolution down to one thousand one hundred fifty-two by seven hundred sixty-eight.

Tom: That’s the trade off; quality versus streamability, and it shows they are thinking about real deployment constraints right from the start.

Jane: And they pointed out that while it's fast and pose free, you still have to be careful about how far away your novel view is from where you started, because image quality drops as the target viewpoint moves further away.

Lu: The authors also mentioned their future work, focusing on online pose refinement and lightweight bundle adjustment to make that geometric consistency even stronger over time.

Tom: So the future direction is definitely about making it more robust and accurate for longer video sequences by adding those refinement tools they mentioned.

Jane: This paper, LiveStre4m: Feed-Forward Live Streaming of Novel Views from Unposed Multi-View Video, gives us a really tangible tool for getting photorealistic novel viewpoints quickly in a streaming environment.

Lu: It’s a strong foundation because it proves that feed forward reconstruction can handle this task efficiently without the heavy computational overhead of traditional methods.

Meng: For me, it means we can start prototyping these features in real-time applications much sooner, rather than waiting for massive optimization pipelines to finish running.

Lalam: I’m confident that this approach is solid; the combination of pose prediction and spatial geometry reconstruction provides a very direct path to novel views when you only have unposed inputs.

Tom: We’ve seen how they managed to do this with just two synchronized input streams at one thousand twenty-four by seven hundred sixty-eight resolution in zero point zero seven seconds per frame on a single GPU.

Jane: It’s a real step toward making novel view synthesis practical for live applications because it cuts out the need for those slow structure from motion pipelines entirely.

Lu: We're moving away from needing perfect ground truth camera parameters for every input, which is a huge win when you don't have that kind of data available.

Meng: From a production standpoint, if we can get this kind of performance on a single GPU for one thousand twenty-four by seven hundred sixty-eight resolution, that drastically lowers the barrier for deploying novel view synthesis features in real applications.

Tom: It’s about making these kinds of complex visual tasks accessible by focusing on feed forward models instead of heavy optimization loops.

Jane: So, to wrap up this discussion on LiveStre4m: Feed-Forward Live Streaming of Novel Views from Unposed Multi-View Video, the authors have shown a way to do real time novel view synthesis from sparse unposed multi-view video inputs using just two synchronized input streams at one thousand twenty-four by seven hundred sixty-eight resolution in zero point zero seven seconds per frame.

Lu: They achieved this by combining a multi view vision transformer for keyframe three dimensional scene reconstruction with a diffusion transformer interpolation module that ensures temporal consistency and stable streaming.

Meng: It’s a significant step toward deployable live novel-view synthesis systems because they’ve managed to produce photorealistic novel viewpoints from sparse unposed input views at one thousand twenty-four by seven hundred sixty-eight resolution in only zero point zero seven seconds per frame on a single GPU.

Tom: We’ll keep an eye on those future work plans for the online pose refinement part; that should really nail down geometric consistency over longer sequences.

Jane: Next time, we look at StyleComposer, because it’s training-free style composition and it’s interesting how they route every style attribute through a representation.

Pedro Quesado Erkut Akdag, Yasaman Kashefbahrami, Willem Menu Egor Bondarev

Eindhoven University of Technology

cs.CV

Submitted: 2026-04-08

Updated: 2026-04-08

Journal ref: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2026, pp. 279-288

Code: https://github.com/pedro-quesado/LiveStre4m

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 87/100

The gist: LiveStre4m proposes a novel viewpoint video live-streaming method that enables real-time Novel View Synthesis from unposed multi-view video inputs by combining a Camera Pose Predictor…

Key concepts

Camera Pose Predictor
This component estimates the exact position and orientation of the camera capturing an image directly from unposed RGB video. It uses Vision Transformers (ViTs) to analyze the input images without needing prior calibration data, allowing the system to work in a fully pose-free manner.
Spatial Module
This module uses transformer architectures to understand the scene's 3D structure and appearance. It takes camera-centric representations and maps them into a global scene token vector, which is then used to predict the parameters for a 3D Gaussian Splatting representation of the scene.
Neural Interpolation Network (NIN)
The NIN ensures temporal consistency in the novel view video. It uses Diffusion Transformers to interpolate between keyframes and includes a high-speed single-image super-resolution module to increase image resolution, resulting in smoother and sharper output frames.

Terminology

Summary

LiveStre4m proposes a novel viewpoint video live-streaming method that enables real-time Novel View Synthesis from unposed multi-view video inputs by combining a Camera Pose Predictor, a Spatial Module, and a Neural Interpolation Network

Core Methodology

LiveStre4m is structured around three key modules: the Camera Pose Predictor, the Spatial Module, and the Neural Interpolation Network (NIN). The Camera Pose Predictor estimates camera parameters directly from unposed RGB images using multi-view ViTs. p. 1

Where the paper says it

LiveStre4m combines three key components: a Camera Pose Predictor, a Spatial Module, and a Neural Interpolation Network (NIN). The Camera Pose Predictor, inspired by previous methods on pose regression and scene reconstruction [27, 28, 41], employs multi-view ViTs to estimate camera parameters directly from unposed RGB images.

The Spatial Module leverages a transformer-based architecture to estimate scene geometry and appearance to generate a 3D Gaussian Splatting representation. p. 1
Where the paper says it

The Camera Pose Predictor, inspired by previous methods on pose regression and scene reconstruction [27, 28, 41], employs multi-view ViTs to estimate camera parameters directly from unposed RGB images. The Spatial Module leverages a transformer-based architecture to estimate scene geometry and appearance to generate a 3D Gaussian Splatting representation.

The NIN module applies Diffusion Transformers for temporal frame interpolation alongside a high-speed single-image super-resolution module. p. 1
Where the paper says it

Although this enables efficient photorealistic NVS, computational costs limit image resolution and video frame rate. The NIN module alleviates this by applying Diffusion Transformers [18] for temporal frame interpolation alongside a high-speed singleimage super-resolution module, enabling temporal consistency, sufficient frame rate and improved image resolution.

Data Flow and Temporal Consistency

The data flow involves predicting camera poses from unposed input RGB views, which are then processed by the Spatial Module to generate a low-resolution novel viewpoint of the scene. After two keyframes are processed, NIN interpolates and increases image resolution to generate an intermediate frame. p. 3

Where the paper says it

Then, the images and predicted camera information are processed by the Spatial Module to generate a low-resolution novel viewpoint of the scene. After two keyframes are processed, NIN interpolates and increases image resolution to generate the high-resolution video snippet.

Finally, all three frames are fed into the Super Resolution module to generate high-resolution frames for the novel view video stream. p. 4
Where the paper says it

This module is a lightweight pretrained CNN architecture [1] performing fast image super-resolution. The resulting high-resolution frames are used to compose the novel view video stream of LiveStre4m.

Key Architectural Components

The Camera Pose Predictor employs a coarse-to-fine strategy to estimate camera poses from tokenized input frames, refining them through attention heads. The Spatial Module uses local view tokens processed by a transformer module Floc(·) to create camera-centric representations, which are then mapped by Fglobal(·) to a single global scene token vector. p. 3

Where the paper says it

To enhance robustness against minor inaccuracies in camera pose predictions, explicit geometric projections are avoided [41]. Instead, a transformer module Fglobal(·) learns to map local point tokens into a single global scene token vector, guided by the predicted camera poses.

This global geometry is upsampled by the Dense Prediction Transformer (DPT) Global Head to predict 3D Gaussian parameters centered on the predicted point map. p. 4
Where the paper says it

By combining appearance features extracted and the predicted geometry, a DPT head predicts 3D Gaussian parameters centered on the predicted pointmap to achieve photorealistic NVS. form a dense 3D point map.

Performance and Efficiency Gains

LiveStre4m attains an average reconstruction time of 0.07s per-frame at 1024 × 768 resolution, significantly outperforming optimization-based dynamic scene representation methods by orders of magnitude in runtime. p. 0

Where the paper says it

LiveStre4m attains an average reconstruction time of 0. 07s per-frame at 1024 × 768 resolution, outperforming the optimization-based dynamic scene representation methods by orders of magnitude in runtime.

The method operates end-to-end, taking sparse unposed input video streams and producing novel viewpoint video in real-time. p. 1
Where the paper says it

The NIN module alleviates this by applying Diffusion Transformers [18] for temporal frame interpolation alongside a high-speed singleimage super-resolution module, enabling temporal consistency, sufficient frame rate and improved image resolution. LiveStre4m operates end-to-end, taking sparse unposed input video streams and producing novel viewpoint video at 1024 × 768 resolution in 0.

Experimental Validation

Experiments on the Neural3DVideo dataset show that LiveStre4m achieves competitive rendering quality compared to state-of-the-art methods, attaining a PSNR of 21.11 at 1024 × 768 resolution. p. 0

Where the paper says it

LiveStre4m attains an average reconstruction time of 0. 07s per-frame at 1024 × 768 resolution, outperforming the optimization-based dynamic scene representation methods by orders of magnitude in runtime.

Furthermore, the method demonstrates a substantial advantage in efficiency, with a runtime of 0. p. 6
Where the paper says it

65 compared to 16. In terms of efficiency, LiveStre4m demonstrates a substantial advantage with a runtime of 0.

074s per frame, over 3. p. 6
Where the paper says it

In terms of efficiency, LiveStre4m demonstrates a substantial advantage with a runtime of 0. 074s per frame, over 3.

3× faster than FLARE. p. 6
Where the paper says it

074s per frame, over 3. 3× faster than FLARE (0.

The performance on the VRU-Basketball dataset also confirms its robustness in highly dynamic scenarios. p. 6
Where the paper says it

We present additional results in supplementary material, including detailed per-scene analysis (Sec. A), an evaluation on the highly dynamic VRUBasketball [32, 33] dataset in (Sec.

Ablation Insights

Ablation studies indicate a trade-off between visual quality and streamability, showing that maintaining frame rates above > 10 fps requires limiting the image resolution to 1152 × 768. The Spatial Module is identified as the component dominating the runtime with 52. p. 7

Where the paper says it

Runtime of Model Components: Table 6 breaks down the computational cost of each component, highlighting potential bottlenecks. The Spatial Module dominates the runtime with 52.

1 ms. The Interpolation Net is significantly lighter at 19. p. 7
Where the paper says it

The Spatial Module dominates the runtime with 52. The Interpolation Net is significantly lighter at 19.

3 ms, validating the choice of interpolating intermediate frames for a more efficient model. p. 7
Where the paper says it

The Interpolation Net is significantly lighter at 19. 3 ms, validating the choice of interpolating intermediate frames for a more efficient model.

The Super Res Net is a lightweight 3-layer CNN with 32 channels pretrained for 2× image upsampling. p. 5
Where the paper says it

The interpolation component of the NIN is a pretrained 12-block DiT with spatial and temporal attention coupled with a 4-block decoder that refines image tokens to generate the interpolated frame. The super-resolution module is a lightweight 3 layer CNN with 32 channels pretrained for 2× image upsampling.

The analysis also shows that increasing the number of output views reduces image quality, as target viewpoints deviate further from the central viewpoint. p. 7
Where the paper says it

Table 5 explores how the number of output views affects the runtime and NVS quality. Results indicate that increasing the number of output images not only increases the runtime, but also reduces image quality, as target viewpoints deviate further from the central viewpoint.

Conclusion and Future Work

LiveStre4m represents a significant step toward livestreaming of novel-viewpoint video in real-time. Potential enhancements include incorporating online pose refinement and lightweight bundle adjustment to improve geometric consistency and visual fidelity. p. 7

Where the paper says it

Discussion LiveStre4m prioritizes efficiency, trading some visual quality for significantly faster inference compared to optimization-based approaches, but to close this gap, potential enhancements are on exploration. Incorporating online pose refinement and lightweight bundle adjustment could improve geometric consistency and visual fidelity.

Scaling multi-view video training data is expected to further enhance generalization and output quality. p. 7
Where the paper says it

Incorporating online pose refinement and lightweight bundle adjustment could improve geometric consistency and visual fidelity. Furthermore, scaling multi-view video training data is expected to further enhance generalization and output quality.

The paper concludes that the method produces photorealistic novel viewpoints from sparse unposed input views at 1024×768 resolution in only 0. p. 7
Where the paper says it

NIN ensures temporal consistency, better image resolution and higher frame rates. Experimental results demonstrate that the proposed method produces photorealistic novel viewpoints from sparse unposed input views at 1024×768 resolution in only 0.

07s per frame on a single GPU. p. 7
Where the paper says it

Experimental results demonstrate that the proposed method produces photorealistic novel viewpoints from sparse unposed input views at 1024×768 resolution in only 0. 07s per frame on a single GPU.

Acknowledgments

This work was supported by the ELEVATION Xecs 23 project. The references section lists numerous works supporting various aspects of the proposed method, such as NeRF [16], Gaussian Splatting [10], and various feed-forward NVS approaches like DUSt3R [29] and FLARE [41]. p. 4

Where the paper says it

Furthermore, this allows LiveStre4m to be applied to various input frame rates while preserving temporal consistency in the generated novel viewpoint video. Training Strategy The proposed method is finetuned with visual losses only, i.

The detailed quantitative comparisons are presented in Tables 1, 2, 3, and 8. p. 5
Where the paper says it

To ensure fair comparison with methods that rely on ground-truth camera parameters, we follow the evaluation protocol established by prior pose-free approaches [5, 30, 38]. Table 1 presents a detailed quantitative comparison of dynamic scene reconstruction methods evaluated on the Neural3DVideo [14] and MeetRoom [13] datasets.

The qualitative results are illustrated in Figures 1 through 6. p. 6
Where the paper says it

Moreover, LiveStre4m performs NVS at twice the output image resolution due to the Super Resolution module. Qualitative results are illustrated in Fig.

Detailed per-scene analysis and pose prediction accuracy metrics are provided in Sections A, B, and C respectively. p. 2
Where the paper says it

An overview of LiveStre4m data flow is first provided, followed by detailed descriptions of the Camera Pose Predictor, Spatial Module, and Neural Interpolation Network (NIN) in Sections 3. 4, respectively.

The paper is available on arXiv as 2604. p. 0
Where the paper says it

For instance, DUSt3R [29] and MASt3R [12] leverage pre-trained Vision Transformers (ViTs) [4] to provide dense geometric correspondences and scene structure, reducing the reliance on conventional optimization-based pipelines. Building on 1 arXiv:2604.

06740. The code is available at github. p. 0
Where the paper says it

These results demonstrate that LiveStre4m makes real-time NVS streaming feasible in practical settings, marking a substantial step toward deployable live novel-view synthesis systems. Code available at: https: //github.

com/pedro-quesado/LiveStre4m. p. 0
Where the paper says it

Code available at: https: //github. com/pedro-quesado/LiveStre4m 1.

The paper is published in the context of arXiv:2604. p. 0
Where the paper says it

For instance, DUSt3R [29] and MASt3R [12] leverage pre-trained Vision Transformers (ViTs) [4] to provide dense geometric correspondences and scene structure, reducing the reliance on conventional optimization-based pipelines. Building on 1 arXiv:2604.

06740. The references section contains citations ranging from [1] to [39]. p. 8
Where the paper says it

Acknowledgments This work was supported by the ELEVATION Xecs 2023022 project References [1] Guillaume Berger, Manik Dhingra, Antoine Mercier, Yashesh Savani, Sunny Panchal, and Fatih Porikli. Quicksrnet: Plain single-image super-resolution architecture for faster inference on mobile platforms, 2023.

The supplementary material includes detailed results and additional evaluations. p. 6
Where the paper says it

4, comparing reference renderings with LiveStre4m outputs across different test scenes, showcasing photorealistic feed-forward 3D scene reconstruction. We present additional results in supplementary material, including detailed per-scene analysis (Sec.

The paper is a substantial step toward deployable live novel-view synthesis systems. p. 0
Where the paper says it

07s per-frame at 1024 × 768 resolution, outperforming the optimization-based dynamic scene representation methods by orders of magnitude in runtime. These results demonstrate that LiveStre4m makes real-time NVS streaming feasible in practical settings, marking a substantial step toward deployable live novel-view synthesis systems.

The runtime comparison highlights LiveStre4m's speed relative to 4DGS and IGS. LiveStre4m achieves competitive PSNR to FLARE on the Neural3DVideo dataset. p. 6
Where the paper says it

As shown in Table 2, LiveStre4m achieves competitive rendering quality compared to state-of-the-art methods. On the Neural3DVideo dataset, LiveStre4m attains a PSNR of 21.

The comparison with FLARE shows LiveStre4m operates in 0. p. 6
Where the paper says it

Although FLARE is not developed for dynamic scenes, it is capable of zero-shot camera-free NVS from sparse input views. For comparison, FLARE is evaluated with the same pipeline as LiveStre4m at each each timestep and its performance is averaged for the entire video.

14 seconds per frame, approximately 55× faster than FLARE. p. 1
Where the paper says it

Although this approach represents a step toward streamable NVS, the optimization requires 12 seconds per frame, which is far too slow for real-time applications. Building upon this idea, IGS [36] is proposed as an even faster approach, reducing reconstruction time to approximately 2.

The paper addresses the challenge of ground-truth camera parameters by proposing a pose-free approach. p. 5
Where the paper says it

For consistent reporting, runtime is defined as the time required to generate a novel viewpoint of a complete video, averaged by the number of frames, excluding pre-processing. To ensure fair comparison with methods that rely on ground-truth camera parameters, we follow the evaluation protocol established by prior pose-free approaches [5, 30, 38].

The authors acknowledge support from the ELEVATION Xecs 23 project. p. 8
Where the paper says it

Acknowledgments This work was supported by the ELEVATION Xecs 2023022 project References [1] Guillaume Berger, Manik Dhingra, Antoine Mercier, Yashesh Savani, Sunny Panchal, and Fatih Porikli. Quicksrnet: Plain single-image super-resolution architecture for faster inference on mobile platforms, 2023.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Improved AI system can perform real-time novel view synthesis from sparse unposed multi-view video inputs using only two synchronized input streams at 1024 x 768 resolution in 0.07 seconds per frame, which is orders of magnitude faster than optimization-based methods.

  2. The system can achieve zero shot NVS with a transformer based Spatial Module guided by camera parameters predicted directly from RGB images, removing the reliance on known camera calibration information to reconstruct scene geometry and appearance.

  3. Temporal consistency in novelview video generation is ensured by a Neural Interpolation Network that applies Diffusion Transformers for temporal frame interpolation alongside a high speed single image super resolution module, resulting in improved image quality and sufficient frame rate.

  4. The system can operate in a fully pose free manner for novel view synthesis, which is crucial because reliable calibration data is unavailable and acquiring these parameters requires either specialized hardware or computationally expensive structure-from-motion pipelines.

  5. The system can be adapted for dynamic scenes by using a feed forward method that enables online per-frame optimization ideas, although the current focus remains on static camera assumptions to maintain real time streaming.

Abstract

Live-streaming Novel View Synthesis (NVS) from unposed multi-view video remains an open challenge in a wide range of applications. Existing methods for dynamic scene representation typically require ground-truth camera parameters and involve lengthy optimizations (about 2.67 s), which makes them unsuitable for live streaming scenarios. To address this issue, we propose a novel viewpoint video live-streaming method (LiveStre4m), a feed-forward model for real-time NVS from unposed sparse multi-view inputs. LiveStre4m introduces a multi-view vision transformer for keyframe 3D scene reconstruction coupled with a diffusion-transformer interpolation module that ensures temporal consistency and stable streaming. In addition, a Camera Pose Predictor module is proposed to efficiently estimate both poses and intrinsics directly from RGB images, removing the reliance on known camera calibration information. Our approach enables temporally consistent novel-view video streaming in real-time using as few as two synchronized unposed input streams. LiveStre4m attains an average reconstruction time of 0.07 s per-frame at 1024 times 768 resolution, outperforming the optimization-based dynamic scene representation methods by orders of magnitude in runtime. These results demonstrate that LiveStre4m makes real-time NVS streaming feasible in practical settings, marking a substantial step toward deployable live novel-view synthesis systems. Code available at: https://github.com/pedro-quesado/LiveStre4m

Sources

Related papers