LVS: Local View Synthesis from Relative Camera Pose by Reusing Previous Views

The gist: This work proposes a per-scene framework for local view synthesis that reuses rendered RGB-D images through relative-pose-guided warping and residual appearance correction, which offers an alternative to conventional methods by avoiding repeated scene rendering.

How it works

1. The core idea is to replace repeated scene rendering for nearby views with relative-pose-guided RGB-D image reuse Our method reuses a rendered RGB-D image through relative-pose-guided warping and RGB residual correction >

2. Geometric warping uses depth and relative pose to transport source content Geometric warping uses depth and relative pose to transport observed structure >

3. A lightweight multiscale network predicts RGB residuals to correct artifacts and infer missing appearance A lightweight multiscale network predicts RGB residuals to correct artifacts and infer missing appearance >

Relative-Pose-Guided Reprojection

The process begins with establishing correspondences using geometric transport, where a source pixel projects to a target pixel according to the equation derived from relative pose Source colors are bilinearly splatted and filtered by depth, and accepted colors are averaged with their splat weights Coverage C stores accumulated weights clipped to [0, 1], with projected centers set to one > Coverage C stores accumulated weights clipped to [0, 1], with projected centers set to one > The warped image is calculated as Iw = W(I1, D1, K, p), where W denotes this fixed geometric procedure

Multiscale Residual Refinement

The framework then refines the geometric warp by predicting an RGB residual R using a multiscale decoder The decoder combines cached source features and geometric cues to predict an RGB residual The prediction is formulated as Ib2 = Iw + R, where R = Dec(F1, I1, Iw, ∆W, C, p) This residual is added to the warped image to obtain the target prediction >

Training and Source Reuse

The encoder and decoder are jointly trained by minimizing a Charbonnier loss L = meanq (Ib2 − I2) 2 + ϵ 2 The training loss minimizes the difference between the predicted and ground-truth target RGB images > For inference, source features and prepared geometry are cached once per source, meaning each target requires reprojection, residual decoding, and composition

Experimental Results

The experiments evaluate reconstruction quality using PSNR and SSIM across different domains like GS-render (three Mip-NeRF 360 scenes) and Blender Table 1 shows that Ours achieves the highest PSNR and SSIM among the evaluated methods in each domain > On GS-render, it exceeds AdaMPI32 by 1.00 dB PSNR, supporting reuse of an existing rendering for nearby target views > The query latency for Ours is reported as 0.95–0.98 ms with a 14.4 MiB source cache

Component Ablation and Limitations

Ablation studies on GS-render demonstrate that residual refinement improves Pure Warp by 0.721 dB PSNR, with a smaller LPIPS change However, performance declines with larger viewpoint changes; Ours has higher PSNR at smaller angles, but 3DGS overtakes it at 15◦ This crossover illustrates the limited viewpoint range of single-source reuse The paper concludes that extending the supported range and enabling cross-scene generalization remain future work >

The framework presents a per-scene approach for local view synthesis by reusing rendered RGB-D images without rerendering the scene. The relative-pose-guided warping preserves shared content, while multiscale residual refinement corrects transport errors and infers missing appearance. These findings highlight image reuse as a practical approach to responsive scene navigation, with potential AR/VR applications >

--- Page 1 ---

LVS: LOCAL VIEW SYNTHESIS FROM RELATIVE CAMERA POSE BY REUSING PREVIOUS VIEWS Qizhou Huo, Xuan Sun, Yongfei Guo, Zhipeng Wang, Yuanhao Gong 3DGS Output = Render(G) Image 1 Image 2 Independent Camera pose p1 Camera pose p2 Ours Output =∆෢I Warp + residual Relative pose Δp Fig. 1. Local view synthesis. Our method reuses a rendered RGB-D image through relative-pose-guided warping and RGB residual correction >

--- Page 2 ---

SOURCE CACHE RESIDUAL DECODER RGB I1 Encoder Cache F1 [8C × H 4 × W 4] c Î2 = Iw + R INPUT GEOMETRIC TRANSPORT OUTPUT & SUPERVISION Depth D1 Reproject Target pose p intrinsics K Iw ΔW = Iw − I1 + Prediction Î2 Target I2 Loss Charbonnier Features Geometry Fused Residual Loss sub + add c concat ↑2×bilinear [C×H×W] tensor C, p Embedding [10C × H 4 × W 4] 1× [C×H×W] RGB residual R ¼ × [4C × H4 × W 4] Full-resolution I1,Iw, ∆W, CI1 ↑2× c ↑2× ½ × [2C × H2 × W2] [2C × H4 × W 4] Fig. 2. Framework overview. Depth and relative pose guide warping, while a multiscale decoder combines cached source features and geometric cues to predict an RGB residual > learns flow, masks, and inpainting for RGB-D synthesis [15]; NeuralPassthrough combines depth-based warping with twoview fusion [16]; Layered inpainting completes hidden content for single-image view synthesis [17, 18] Reusing a single rendered image therefore requires preserving transported detail while correcting regions where correspondence is unreliable or absent >

--- Page 3 ---

GT AdaMPI32 CheapNVS Warp-only Network-only Ours Target Source Counter · 3.5° 25.31 / 0.841 18.09 / 0.518 Target Kitchen · 15° 20.19 / 0.578 Kitchen · 17.64 / 0.485 Prediction Error Prediction Error Fig. 3 Qualitative comparison on Counter (3.5◦) and Kitchen (15◦) Scores are PSNR/SSIM

--- Page 4 ---

Table 1 Test quality and query cost Cache and peak memory are in MiB; 3DGS parameter counts refer to stored Gaussian scalars Best values are bold Query (ms) Ours AdaMPI32 CheapNVS 3DGS

--- Page 5 ---

Table 2 Component ablation on GS-render, averaged over three scenes Best values are bold Variant PSNR ↑ SSIM ↑ LPIPS ↓ Params (K) Warp-only 26.166 0.8340 0.1898 Network-only 19.659 0.5833 0.4299 Residual-Full 26.887 0.8407 0.1888 Pure Network performs substantially worse, indicating the value of depth-guided transport in this system >

--- Page 5 ---

Figure 4 Viewpoint sensitivity on Blender Ours has higher PSNR at smaller angles, but 3DGS overtakes it at 15◦ This crossover illustrates the limited viewpoint range of single-source reuse >

--- Page 4 ---

Table 1 reports query latency of 0.95–0.98 ms for Ours, with a 14.4 MiB source cache and 95.8 MiB peak memory Source reuse shares feature extraction and geometry preparation across queries; target-dependent reprojection and residual prediction remain necessary for each new view >

--- Page 3 ---

Figure 3 compares identical source–target pairs At 3.5◦, residual refinement reduces errors around object boundaries relative to Pure Warp At 15◦, it improves PSNR while visi-

--- Page 4 ---

Table 2 evaluates geometry and residual refinement on the three GS-render scenes Full refinement improves Pure Warp by 0.

1. textbfbold relative-pose-guided framework for local view synthesis by reusing rendered RGB-D images: Improves interactive scene exploration by replacing repeated scene rendering with relative-pose-guided RGB-D image reuse. This allows for responsive scene exploration in augmented and virtual reality applications without the computational cost of recomputing every target view from scratch.

2. textbfmultiscale residual network with source feature reuse: Enhances reconstruction quality by using a multiscale decoder to predict RGB residuals to correct artifacts and infer missing appearance. This refinement is applied by adding the predicted residual, Ib2 = Iw + R, ensuring that spatial detail is preserved while correcting errors from geometric transport.

3. textbfcached source features: Reduces query latency and computational load: The system employs Cached source features further reduce repeated computation, meaning that once source features are extracted, each target view only requires reprojection, residual decoding, and composition. This supports low query latency of 0.95–0.98 ms in Table 1 for the proposed method.

4. textbfgeometry and residual refinement separation: Improves robustness against geometric errors: The framework explicitly separates geometric transport from residual appearance correction, allowing the system to utilize depth and relative pose to transport source content while using the network to correct inaccuracies, as shown by how residual refinement improves PSNR by 0.72 dB over pure warping.

Interactive scene exploration requires frequent view updates, although small camera motions preserve much of the visible content. Conventional 3D Gaussian Splatting nevertheless renders each target view, leaving this image overlap unexploited. Reusing rendered images offers an alternative. Geometric warping alone cannot recover newly exposed content and remains sensitive to depth errors. We propose a per-scene framework that replaces repeated scene rendering for nearby views with relative-pose-guided RGB-D image reuse. Geometric warping uses depth and relative pose to transport source content, while a lightweight multiscale network predicts RGB residuals to correct artifacts and infer missing appearance. Cached source features further reduce repeated computation. On GS-render, residual refinement improves PSNR by 0.72~dB over pure warping; evaluations on captured and rendered scenes demonstrate low query latency. This separation of scene rendering from local view updates supports responsive scene exploration, with potential applications in augmented and virtual reality.