LoC-IN: Real-Time Localization of Cameras in Neural Rendered Prior LiDAR Maps

arXiv:2511.16349 · cs.CV, cs.GR · Submitted 2025-11-20 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "LoC-IN: Real-Time Localization of Cameras in Neural Rendered Prior LiDAR Maps".

Paper summary: Tom: To get into how they’re doing it, they present two main approaches: Online Render and Match, or R andM. This method uses real-time neural rendering to synthesize a virtual image based on the camera's last known position, and then matches that synthetic view with the next live frame to figure out the current six degrees of freedom pose.

Jane: And then they also have Prebuild and Localize, or P andL. This approach involves building a feature map from those synthetic point cloud renders offline so you can use standard real-time SLAM pipelines for tracking with almost no manual work needed during the actual tracking phase.

Lu: The foundation for both of these methods is a neural point cloud renderer. They design this to project three dee points into a 2D image plane using a hierarchical depth filter to clean up background points leaking into the foreground, and then use a U-Net to fill in those holes in the images <ref:2511.16349#pg1>.

Meng: From an engineering standpoint, they mention this rendering pipeline has three passes: a depth pass for minimum depth, color accumulation and averaging for points on similar depths, and then filtering out background leakage with that hierarchical filter followed by the U-Net hole-filling network.

Lalam: The authors say this design eliminates the need for frustum culling overhead and synchronization locks, which they claim results in a speedup of about one point seven five times with visual quality staying identical for a ninety six zero by seven two zero resolution and a point cloud of fifty-eight million points.

Tom: That’s some solid performance data, but the core idea is using this neural renderer to establish those 2D to three dee correspondences between the images and the point cloud, which they say improves matching compared to older methods like ORB features <ref:2511.16349#pg1>.

Jane: And they mention that for feature extraction and matching in the Online Render and Match pipeline specifically, they chose XFeat because those deep neural features are more robust to changes in viewpoint or lighting compared to something like ORB.

Lu: The P andL method focuses on offline mapping where keyframes and landmarks are rendered using the camera's intrinsic properties, which includes things like the intrinsic matrix R(P, Tj, Kcam), and then they extract ORB features from those renders to associate them with their corresponding three dimensional landmarks.

Meng: For me, I’m looking at this P andL idea because it sounds efficient for deployment—precomputing a map offline means the real-time tracking part only has to run the standard SLAM pipeline without having to do heavy neural rendering every single frame.

Conclusion: Tom: So, looking at "LoC-IN: Real-Time Localization of Cameras in Neural Rendered Prior LiDAR Maps," the paper presents a method that achieves drift-free localization and correct metric scale by using neural rendering to bridge the gap between synthetic views from a point cloud and real camera frames.

Jane: The two main ways they present this are the Online Render and Match approach for real-time tracking, and the Prebuild and Localize approach which lets you create a map offline for traditional SLAM pipelines.

Lu: What this means in practice is that you get localization that is consistent within a single global coordinate system derived from the LiDAR point cloud right from the beginning, instead of having to correct for drift later.

Meng: For deployment, if you choose P andL, it sounds like a way to get high accuracy without needing massive real-time GPU power for every frame; you just need that offline precomputation done once.

Lalam: The authors provide ground-truth datasets on their own to back up these methods, and they show that the R andM variant is more robust than both the ScanNet++ dataset and their own P andL poses, while P andL gets higher SSIM scores for photometric alignment.

Tom: It seems like this work directly addresses the drift and scale ambiguity problems that have plagued camera localization in robotics for a long time by introducing this neural rendering technique.

Jane: The overall implication is that you can get reliable, metric-scale tracking using existing LiDAR data even if you don't have perfect fiducials or loop closures available during the operation.

Joni Vanherck, Steven Moonen, Brent Zoomers, Kobe Werner, Jeroen Put Lode Jorissen, Nick Michiels

Hasselt University - Digital Future Lab - Flanders Make

cs.CV, cs.GR

Submitted: 2025-11-20

Updated: 2026-11-16

Comments: Published in IEEE Robotics and Automation Letters, 8 pages, 4 figures

Journal ref: IEEE Robotics and Automation Letters (Volume: 11, Issue: 12, pp:14980-14987, December 2026)

DOI: 10.1109/LRA.2026.3748685

Code: https://github.com/MichaelGrupp/evo

License: http://creativecommons.org/licenses/by-nc-nd/4.0/

Importance score: 79/100

The gist: Accurate camera localization is crucial for robotics and Extended Reality (XR), enabling reliable navigation and alignment of virtual and real content By rendering synthetic views from a…

Key concepts

Neural Point Cloud Renderer
This is a network designed to turn 3D LiDAR points into realistic 2D images. It uses a hierarchical depth filter to clean up background noise and a U-Net to fill in missing image details, ensuring the generated synthetic views look high-quality and accurate for camera matching.
Online Render & Match (R&M)
This method tracks a camera live by using neural rendering. It takes the last known camera position, renders what that view *should* look like from the LiDAR map, and then tries to match this synthetic image to the current live camera frame to find its new location.
Prebuild & Localize (P&L)
This approach maps a LiDAR point cloud offline first. It uses this map to create a drift-free SLAM system where keyframes and landmarks are positioned precisely using ground truth from the laser scan, allowing for efficient real-time tracking later.

Terminology

Summary

Accurate camera localization is crucial for robotics and Extended Reality (XR), enabling reliable navigation and alignment of virtual and real content By rendering synthetic views from a pre-captured, highly accurate colored LiDAR point cloud, this work introduces a realtime method for localizing a camera within the point cloud The result is drift-free camera tracking with correct metric scale in the global LiDAR coordinate system

Core Methodology

The paper introduces two primary methods for camera localization within a point cloud: (1) Online Render & Match (R&M) and (2) Prebuild & Localize (P&L) The R&M method uses real-time neural rendering to synthesize a virtual image from the last known pose, which is then matched with the next camera frame Conversely, the P&L method precomputes a SLAM map offline, enabling efficient, real-time tracking using standard SLAM pipelines Both approaches rely on a neural renderer for high-fidelity LiDAR point clouds

Neural Point Cloud Renderer

The neural point cloud renderer is designed to project 3D points into a 2D image plane using a hierarchical depth filter to remove background points leaking into the foreground, and employs a U-Net to fill gaps in the image The pipeline involves three passes: depth pass, color accumulation, and averaging to compute per-pixel color Background leakage is removed by a hierarchical depth filter [35], followed by a U-Net hole-filling network U This design eliminates the frustum-culling overhead and synchronization locks, resulting in a 1.75× speedup with identical visual quality for a resolution of 960×720 and a point cloud of 58M points

Online Render & Match Pipeline

The Online Render & Match method performs absolute camera localization by matching the live camera feed to synthetic views generated from the LiDAR point cloud This pipeline involves preprocessing, relocalization, and tracking stages During tracking, the previous pose is used to render a synthetic view, which is matched to the live camera frame to estimate the current 6DoF pose For feature extraction and matching in this pipeline, XFeat [25] is chosen because the deep neural features are more robust to viewpoint and lighting changes compared to ORB [29]

Prebuild & Localize Pipeline

The Prebuild & Localize method splits the process into mapping and localization, creating a map directly from the LiDAR point cloud offline For mapping, cubemap-like images/keyframes are rendered using the intrinsic properties of the camera (Ij, Dj) = R(P, Tj, Kcam) ORB features are extracted from these renders and associated with their corresponding 3D landmarks The resulting SLAM map is drift-free, as each keyframe and landmark is positioned using groundtruth poses obtained directly from the laser scan

Evaluation Results

The methods were evaluated on the ScanNet++ dataset and two custom ground-truth datasets The results show that the R&M variant is more robust, successfully localizing more images than both ScanNet++ and our P&L poses In contrast, the P&L variant achieves the highest SSIM scores, indicating better photometric alignment The Prebuild & Localize method constructs a drift-free SLAM map offline, which can be used by traditional SLAM pipelines without modification

Conclusion and Future Directions

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Neural rendering technique adaptation for domain gap reduction: The method adapts a neural rendering technique to mitigate artifacts such as holes due to occlusions and limited point cloud density, or artifacts such as background information leaking into the foreground, both of which interfere with reliable feature extraction. This allows for drift-free camera tracking with correct metric scale in the global LiDAR coordinate system.

  2. Online Render & Match method implementation: This method enables absolute camera localization by matching the live camera feed to Point cloud, and it achieves this by exploiting correspondences that are derived from the LiDAR-based ground-truth geometry, ensuring no accumulation of drift during tracking.

  3. Prebuild & Localize method for traditional SLAM integration: This approach allows for a precomputed SLAM map offline, enabling efficient, real-time tracking using standard SLAM pipelines without modification, which eliminates the need for online rendering and results in a significantly lower computational load while maintaining real-time, drift-free localization.

Abstract

Accurate camera localization is crucial for robotics and Extended Reality (XR), enabling reliable navigation and alignment of virtual and real content. Existing visual methods often suffer from drift, scale ambiguity, and depend on fiducials or loop closure. This work introduces a real-time method for localizing a camera within a pre-captured, highly accurate colored LiDAR point cloud. By rendering synthetic views from this cloud, 2D-3D correspondences are established between live frames and the point cloud. A neural rendering technique narrows the domain gap between synthetic and real images, reducing occlusion and background artifacts to improve feature matching. The result is drift-free camera tracking with correct metric scale in the global LiDAR coordinate system. Two real-time variants are presented: Online Render and Match (R&M), and Prebuild and Localize (P&L). We demonstrate improved results on the ScanNet++ dataset and outperform existing SLAM pipelines.

Related papers