Learning Which Correspondences to Trust: Confidence-Weighted Event-Camera Localization in LiDAR Maps

arXiv:2610.11967 · cs.CV, cs.RO · Submitted 2026-10-08 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "Learning Which Correspondences to Trust".

Jane: The gist The paper proposes a method called CELL to learn per-correspondence confidence end-to-end through a differentiable probabilistic PnP solver,

Tom: First, who's behind it and why it matters.

Paper summary: Tom: Alright folks, we're diving into this paper today. It’s called "Learning Which Correspondences to Trust: Confidence-Weighted Event-Camera Localization in LiDAR Maps." What they're doing here is taking event cameras and trying to map them onto a pre-built LiDAR map.

Jane: So, the big idea of this work is that instead of just matching things geometrically and hoping for the best, they’re learning which individual correspondences are actually trustworthy. It looks at how much each match contributes to figuring out where the camera is in threeD space.

Lu: Yeah, it addresses a problem where you can't just use the usual way of learning this confidence—which is using pixel errors—because that approach has this bias related to depth.

Tom: What does that mean for us when we're looking at how reliable these matches are?

Meng: It means that small pixel errors tend to happen at larger depths, and those don't actually help you constrain the camera pose much. So, if you only look at the error, it’s misleading because of where the match is located.

Lalam: The paper claims that instead of learning confidence from image-space error targets, they optimize correspondence weights based on how well they constrain the overall pose distribution using a differentiable probabilistic PnP solver.

Tom: That sounds like a shift in focus, moving from just minimizing error to minimizing pose uncertainty with every single match. So what's the actual mechanism behind that?

Jane: They introduce a loss term called the log-partition term inside their PnP solver, and they upweight it with gamma equals one point five to emphasize making sure the resulting pose distribution is sharp.

Lu: That log-partition term is what encourages weight configurations that actually result in a better constrained pose distribution, which feels like a much more direct way to learn reliability than just looking at the error on a pixel.

Meng: So this learned confidence map is then used in three different ways: it reweights the flow supervision during training, it drives selection at test time, and it weights a final edge-matching refinement. It’s pretty integrated.

Tom: That’s where things get interesting for the practical side—how does this actually change how we train these systems? How does decoupling the confidence head from the flow backbone work in practice?

Paper summary: Jane: They use a decoupled training scheme where the pose loss is applied with the flow detached, which means they’re only training that confidence head when it’s not directly affecting the flow or edge branches.

Lu: That concentrates the flow supervision onto those confident, pose-important regions, so you're not polluting the backbone with noise from unreliable matches.

Meng: So we get a cleaner flow signal because of this, which should translate into better event image estimation overall. But what about when we actually use this at test time? How do we deploy it?

Tom: At test time, they use the learned confidence to bias the correspondence set through a stochastic sampling method where you keep a match with probability that increases with its confidence.

Jane: They use floor equal zero point five—so even a correspondence with minimum normalized confidence has a fifty percent chance of being kept, and it rises linearly to one for the most confident ones.

Lu: That selection strategy is designed to be robust against imperfect confidence maps while still favoring the reliable matches that the training process learned.

Meng: And there’s this final refinement step where they weigh an edge-matching refinement using both the edge-probability map and this new confidence, w i equals pi ci. They say it gives a small additional improvement over just using the baseline for weighting.

Tom: So, we're looking at how they handle the input data too. They use a partial depth completion method that is conservative because filling in every single pixel just blurs the object edges, which they want to preserve.

Jane: They keep the rendered depth ID sparse—about twenty percent of pixels—and only do a single dilation with a three by three cross kernel to fill in gaps without hallucinating anything across large regions.

Lu: That partial completion increases the fraction of valid depth pixels from about twenty percent up to sixty eight percent, which provides much more valid correspondences for the flow and pose supervision while keeping those object edges intact.

Meng: So we get better input data quality without introducing those geometrically wrong matches that can happen when you try to fill everything in. But they do mention a limitation here: filling every pixel at once can actually hurt accuracy below even the sparse input, which is why this partial method is better for their metrics.

Paper summary: Tom: It sounds like the whole point of this paper, "Learning Which Correspondences to Trust: Confidence-Weighted Event-Camera Localization in LiDAR Maps," is moving away from simple geometric consensus toward a system that learns and trusts individual correspondences based on how well they constrain the final camera pose.

Jane: The authors are showing that modeling this correspondence reliability end-to-end through a differentiable probabilistic PnP solver yields a better constrained pose distribution than just using the per-correspondence error alone.

Lu: It’s about learning confidence through the pose itself, which then guides both training and inference, leading to gains like reducing median translation error by up to twenty-six point nine percent on M3ED and rotation error by up to fifteen point eight percent on DSEC.

Meng: From an engineering standpoint, the main addition that actually changes things is this confidence-guided selection at test time and the edge-matching refinement weighting. The core network itself stays pretty similar, which keeps runtime and memory usage low.

Tom: So for anyone listening who might be building a system for robotics or autonomous vehicles, this suggests that paying attention to the quality of individual correspondences rather than just their existence can significantly improve localization accuracy when using event cameras with LiDAR maps.

Jane: It means we can have more reliable pose estimates because the system actively knows which matches to trust and which ones to down-weight during critical steps.

Lu: The potential here is that this learned confidence could become a really powerful cultural tool for AI systems, allowing them to make decisions based on a nuanced understanding of data reliability rather than just raw data quantity.

Meng: I see the limitation they flag is that the learned confidence needs enough training data to become reliable, which means its benefit might be smaller in scenes where you don't have a lot of labeled event-LiDAR pairs.

Tom: So while the performance gains on datasets like M3ED and DSEC are pretty solid, we still need more examples to make this confidence learning truly robust across all kinds of real-world environments. That’s what comes next for this research.

Conclusion: Tom: So, we’re wrapping up this deep dive into "Learning Which Correspondences to Trust: Confidence-Weighted Event-Camera Localization in LiDAR Maps." Essentially, they figured out how to let an event camera figure out where it is against a fixed LiDAR map by learning which individual matches are trustworthy.

Jane: They took the standard way of matching flow and depth and added a layer of learned confidence, using a probabilistic PnP solver to figure out how much each match actually constrains the camera's pose.

Lu: The authors show that this approach fixes a problem where simple error matching fails because small pixel errors at large depths don't really tell you anything about the camera’s position.

Meng: For us, as engineers, this means we can stop training our systems on bad data points that are just noisy because they don't contribute much to the final pose estimate.

Lalam: And this learned confidence map gets used in three ways: it tunes how much we trust the flow during training, it filters which matches we keep at test time, and it refines the final edge match with a weight based on both the match’s quality and its confidence.

Tom: So, what does this mean for the people out there who just listen to you? It means that when an event camera is trying to figure out its location in a complex environment with LiDAR, it becomes smarter about which pieces of information it actually uses.

Jane: It moves the focus from just getting a pose estimate to making an informed decision about the reliability of every single piece of data the system sees.

Lu: And Lalam’s vision here is that as AI gets better at this, these systems will become much more robust and less prone to errors caused by bad sensor readings or noisy maps.

Meng: From a practical standpoint, it suggests we can simplify our training pipelines because we don't have to worry as much about the flow network getting confused by low-quality correspondences.

Tom: It’s a big shift from treating all matches equally, which is something everyone in this field has been wrestling with for a long time.

Jane: And while they show huge gains on specific datasets like M3ED and DSEC, they also point out that getting that learned confidence to be really reliable still takes a good amount of training data.

Lu: That’s the trade-off, right? The method is powerful, but it relies on having enough examples for the AI to learn what "trustworthy" looks like in a real scene.

Meng: So for now, it’s a really strong tool for high-quality datasets, but we still need those large amounts of data to make it work everywhere.

Tom: Exactly. It’s a solid step forward in making event-based localization much more precise and realistic than what we could achieve just with the old methods.

Jane: And next time we talk about this, we’ll look at how they handle the depth input, because that partial completion method is actually quite clever.

Panagiotis Kiousis, Kuangyi Chen, Jun Zhang, Friedrich Fraundorfer

Department of Electrical & Computer Engineering, University of Patras, Greece · Institute of Visual Computing, Graz University of Technology

cs.CV, cs.RO

Submitted: 2026-10-08

Updated: 2026-10-08

Comments: 8 pages, 8 figures/tables. Submitted to IEEE ICRA 2027 (under review). Code: https://github.com/panagiotisq/CELL

Code: https://github.com/panagiotisq/CELL

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 84/100

The gist: The gist The paper proposes a method called CELL to learn per-correspondence confidence end-to-end through a differentiable probabilistic PnP solver, which improves event camera localization against

Key concepts

Per-correspondence Confidence Head
A small network attached to the input image predicts a confidence map for every pixel. This map is derived from features like GRU hidden states and motion features. It outputs a logit map, which is converted into a positive confidence map using the softplus function, allowing the system to quantify how reliable each potential 2D-3D match is.
Probabilistic PnP Solver
Instead of relying solely on geometric consensus, this solver optimizes correspondence weights based on how well they constrain the camera pose distribution. It uses a loss function combining a target reprojection error term and a log-partition term that encourages the weighted correspondences to yield a sharper, better-constrained pose estimate.
Decoupled Training Scheme
The confidence head is trained separately from the main flow and edge backbone. After training, this learned confidence map is used to reweight the flow supervision during training. This ensures that the flow network only learns from high-confidence regions, preventing pose gradients from corrupting the primary feature extraction layers.

Terminology

Summary

The gist The paper proposes a method called CELL to learn per-correspondence confidence end-to-end through a differentiable probabilistic PnP solver, which improves event camera localization against LiDAR maps by addressing depth-dependent biases in traditional flow-error supervision.

How it works

The core idea is to move beyond models that rely solely on geometric consensus during pose estimation by explicitly modeling the reliability of individual correspondences The natural way to learn this confidence is through the per-correspondence error to constrain the learning of confidence Instead of matching confidence to an image-space error target, the probabilistic pose objective optimizes the correspondence weights according to how well they constrain the pose distribution This results in a learned confidence that is used in three ways: (i) it reweights the flow supervision in a decoupled training scheme that keeps pose gradients out of the flow/edge backbone (ii) it drives a probabilistic correspondence selection at test time and (iii) together with the network’s edge-probability it weights a final edge-matching refinement.

Partial depth completion

To increase the spatial support of the sparse LiDAR projection while preserving depth discontinuities, the method employs a conservative partial depth completion The rendered depth ID is sparse (approximately 20% of pixels), and aggressive completion may propagate depth across large unsupported regions, blurring depth discontinuities Therefore, it completes the depth only partially—a single dilation of the valid points with a 3 × 3 cross kernel, without filling large gaps This increases the fraction of valid depth pixels from approximately 20% to 68%—supplying the flow and pose supervision with far more valid 2D–3D correspondences—while preserving the object edges the edge branch and later edge matching rely on.

Per-correspondence confidence head

A small head is attached to predict a dense per-pixel confidence map c, which reads three feature groups from the IFR: (i) K=4 snapshots of the GRU hidden state h, taken at fixed iteration indices and concatenated; (ii) the context features Fctx; and (iii) motion features derived from the correlation lookup and current flow These are concatenated into a per-pixel feature vector g and passed through a two-layer 1×1 convolutional MLP The single-channel logit map z is bilinearly upsampled to the input resolution and converted to a positive confidence map: c = softplus(z) > 0 The per-correspondence confidence ci is obtained by sampling c at the pixel of correspondence i.

Learning confidence through the pose

The method learns weights end-to-end through a differentiable probabilistic PnP solver, which is more effective than using image-space error as supervision because pixel-space residuals are depth-dependent The per-correspondence pose weights c˜i = M softmaxi(z) are a softmax of the logit z over the M correspondences sampled in a frame, scaled so that ∑i c˜i = M and the weights average to 1, matching the normalization EPro-PnP’s formulation The pose loss combines a target term and a log-partition term, defined as follows: Ltgt = ∑i c˜iri(Tgt), (3) Lpred = logZ exp−∑i c˜iri(T) dT, (4) Lpose = Ltgt +γ Lpred. (5) The target term Ltgt alone rewards correspondences with low reprojection error, which tend to be distant, low-parallax points—the source of the depth bias The log-partition Lpred is small only when the weighted correspondences make the pose distribution sharp We upweight the log-partition term with γ=1.5 to emphasize poselevel constraint quality, and use the resulting confidence for correspondence selection in Sec. IV-D.

Decoupled training

The confidence head is trained together with the flow/edge network but decoupled from it The pose loss is applied with the flow detached, so it trains only the confidence head (whose inputs are detached); the flow and edge branches keep training on their own losses The same confidence, detached, reweights the flow supervision: L c flow = ∑u,v sgc(u,v) 1[fgt(u,v)!= 0]∥f(u, v)−fgt(u,v)∥2 + ε a small constant for numerical stability This concentrates flow supervision on confident, pose-important regions and improves the flow without the pose gradient entering the backbone.

Confidence-guided selection at test time

Given the learned confidence, one strategy is to bias the correspondence set through confidence-dependent stochastic sampling Each correspondence is retained with a probability that increases with its confidence, qi = floor+ (1−floor) cˆi, (7) With floor=0.5 (denoted floor0.5), even a correspondence with the minimum normalized confidence is retained with probability 0.5, rising linearly to 1 for the most confident This makes the selection robust to imperfect confidence while still favouring the reliable correspondences.

Edge-matching refinement weighting

The final pose is refined by aligning predicted depth-edges to observed event edges using a weighted chamfer distance, where each edge is weighted by both the edge-probability and the confidence, wi = pi ci The combination gives the best accuracy over both the baseline and the non-refined pose. Weighting by both provides a small additional improvement over uniform weighting (+0.29 vs. +0.26 points relative to the non-refined pose), suggesting that the learned confidence can also provide a useful weighting cue for the refinement.

Depth completion

Table V isolates the depth input by comparing three completions on our model trained with the decoupled scheme, showing that a full completion is counterproductive: it fills nearly every pixel (∼99.8%) but hallucinates depth at object margins (Fig. 4), which adds geometrically wrong correspondences—accuracy drops below even the sparse input (50.6 vs. 51.1) and the flow itself degrades The partial completion instead fills conservatively (∼68%): it grows the valid regions enough to add useful correspondences while preserving object boundaries, and is the best of the three on every metric (+6.5 accuracy, −2.4 cm median translation, and lowest EPE).

Incremental ablation

Table VI incrementally adds proposed components on falcon outdoor day penno parking, showing that partial depth completion and decoupled confidence training provide the largest gains The full system reduces the median translation and rotation errors by 3.77 cm and 0.186◦ and improves localization accuracy by 10.4 percentage points over the baseline.

Runtime and memory

The core network is essentially unchanged from the baseline, as the confidence head and partial-depth dilation add negligible runtime and memory over the baseline network The edge-matching refinement is the one substantial addition, being best viewed as an optional final stage. The flow estimation dominates the network and is linear in the iteration count, so halving it (24→12) roughly halves that stage

Conclusion

Together, these components improve localization over the LEAR baseline on both M3ED and DSEC The learned confidence needs sufficient training data to become reliable, which limits its benefit on datascarce scenes, and the gains on DSEC are modest since accuracy there is already near-saturated The method also depends on the initial pose used to render the depth view and may degrade when this pose is far from the true one.

References

[1] K. Chen, J. Zhang, and F. Fraundorfer, “EVLoc: Event-based visual localization in LiDAR maps via event-depth registration,” in Proc. IEEE Int. Conf. Robot. Autom. (ICRA), 2025.

[2] K. Chen, J. Zhang, Y. Hu, Y.-Zhou, and F.-Fraundorfer, “LEAR: Learning edge-aware representations for event-to-LiDAR localization,” in Proc. IEEE Int. Conf. Robot. Autom., 2026.

[3] Z. Teed and J. Deng, “RAFT: Recurrent all-pairs field transforms for optical flow,” in Proc. Eur. Conf. Comput.-Vis., 2020, pp. 402–419.

[4] M. Persson and K.-Nordberg, “Lambda twist: An accurate fast robust perspective three point (P3P) solver,” in Proc. Eur. Conf. Comput.-Vis., 2018.

[5] M. A. Fischler and R. C.

Improvements for AI systems

  1. The system can learn a pose-supervised confidence by using a differentiable probabilistic PnP to optimize correspondence weights based on how well they constrain the pose distribution, rather than relying on image-space error, which is biased by depth.

  2. This learned confidence is used in three ways: "(i) it reweights the flow supervision in a decoupled training scheme that keeps pose gradients out of the flow/edge backbone; (ii) it drives a probabilistic correspondence selection at test time; and (iii) together with the network’s edge-probability it weights a final edge-matching refinement."

  3. The system can improve localization by using confidence-guided selection, specifically employing floor0.5, which retains correspondences with a probability that increases with its confidence, making the selection robust to imperfect confidence while still favouring the reliable correspondences.

  4. The refinement stage can be enhanced by weighting edges using both cues: both (edge-prob × conf), which is shown to provide a gain of +0.32 in accuracy over the baseline, as it combines geometric reliability with learned pose reliability.

  5. The system can increase the number of useful correspondences by employing a partial-completion depth representation, which adds signal without hallucinating across large gaps, resulting in a best overall performance on every metric for depth input compared to sparse or full completion.

Related papers