Sensor2Sensor: Cross-Embodiment Sensor Conversion for Autonomous Driving

arXiv:2605.22809 · cs.CV · Submitted 2026-05-21 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "Sensor2Sensor: Cross-Embodiment Sensor Conversion for Autonomous Driving".

Jane: Here's the gist of the paper. Sensor2Sensor is a novel generative paradigm that translates in-the-wild monocular dashcam videos into a high-fidelity, multi-modal sensor suite (AV logs) comprising multi-view camera images and LiDAR point clouds <ref:2605.22809#pg2>.

Tom: This enables cross-embodiment sensor conversion to maximally leverage real-world long-tail data for AV system validation <ref:2605.22809#pg2>.

Jane: Data Bridging Strategy The paper addresses the lack of paired training data by proposing a novel synthetic data-pairing pipeline <ref:2605.22809#pg2>.

Tom: This pipeline leverages existing AV logs to reconstruct 4D scene representations via dynamic 3D Gaussian Splatting (3DGS) <ref:2605.22809#pg2>.

Title and authors: Tom: So we're talking about Sensor2Sensor today. It’s a new way to take those everyday videos from dashcams or the internet and turn them into actual sensor data logs that are useful for testing self-driving cars in the real world Jane.

Jane: Right, so imagine you have a normal video of someone driving, and instead of just looking at the picture, you get all kinds of sensor information—like multiple camera angles and even LiDAR point clouds—for that exact same moment Tom.

Lu: Essentially, this paper is about bridging that gap between what we see in a single video and what an actual car's sensor suite produces Jane. It’s tackling the problem of needing massive amounts of diverse data for safety testing across different types of vehicles Tom.

Meng: But the big hurdle is getting those paired datasets, right? You don't have perfect videos that already have LiDAR data attached for every single scenario Jane.

Lu: That’s where they propose a synthetic data-pairing pipeline, and it uses something called dynamic three dimensional Gaussian Splatting or three deeGS to reconstruct the scene from existing AV logs Tom <ref:2605.22809#pg1>.

Tom: So they take what we already have—real AV logs—and use that to build a 4D representation of the scene, which is really neat because it helps create something new Jane <ref:2605.22809#pg1>.

Jane: And then from that reconstructed scene, they render synthetic dashcam views using camera parameters sampled from real-world dashcams, which gives them the paired training data of synthetic dashcam videos and real AV logs Lu.

Lu: That process is how they solve the lack of paired training data by creating this required corpus for validation Tom.

Tom: The core of it is a conditional diffusion model that takes that single dashcam video as input and generates both eight different camera images and the LiDAR point clouds simultaneously Jane.

Jane: And they make sure everything stays consistent using cross sensor attention for the image and three dee attention blocks for the multiple camera views Lu <ref:2605.22809#pg1>.

Lu: The LiDAR generation uses a separate VAE that encodes depth, intensity, and elongation into a latent space where it minimizes losses for those components and also includes binary cross entropy loss for the validity mask Tom.

Tom: So they’re not just guessing shapes; they are explicitly training the model to generate plausible three dee geometry with features like range and intensity Jane <ref:2605.22809#pg1>.

Jane: They use a total loss that includes L1 reconstruction losses for those things, plus LPIPS losses on surface normals and elongation to make sure the generated LiDAR looks perceptually good Lu.

Title and authors: Lu: They also introduced cross sensor attention within each U-Net block, which is designed to help the image features and the LiDAR features talk to each other continuously Tom.

Tom: That cross sensor attention is key for making sure the camera view and the point cloud match up correctly, right?

Jane: Exactly. But they also extended this into auto-regressive generation using an algorithm called DAgger to help combat temporal inconsistency when converting third-party videos into driving logs Lu.

Lu: The DAgger algorithm augments the training context by including the model's own generations, which helps improve robustness by mitigating short term error accumulation Tom.

Tom: So they’re not just generating one frame; they are doing this iterative process of generating rollouts and retraining on that new context to make the video sequence more stable Jane.

Jane: It’s an interesting approach because it moves beyond just a single snapshot and tries to model how things evolve over time in a driving scene Lu.

Tom: So, what does this mean for the people who actually use these models? Does it help us get better autonomous systems?

Jane: It means we can use tons of real world driving videos—even those from the internet or dashcams—to train and validate autonomous systems that are tailored to a specific vehicle type Lu.

Lu: The authors show that this method achieves state-of-the-art fidelity in cross embodiment sensor generation, which means it’s converting challenging in-the-wild videos into realistic data formats Tom.

Tom: And they have some solid numbers supporting that, like Frechet Inception Distance for image realism and Frechet Video Distance for video realism Jane.

Jane: Qualitatively, the paper shows that the generated LiDAR point clouds are cleaner than baselines and they maintain cross modal consistency between the image and LiDAR Lu.

Lu: And human evaluations back this up too; it’s top preferred in about eighty-three percent of image cases for dashcam data, and over ninety-four percent in pairwise comparisons against another method called X Drive Tom.

Tom: So, to wrap up the main point—Sensor2Sensor is a novel generative paradigm that bridges the embodiment gap between consumer driving videos and the complex sensor suites needed for autonomous driving validation Jane.

Jane: By using 4DGS based data pairing and a conditional diffusion architecture, they convert monocular third-party videos into synchronized multiview camera streams and LiDAR point clouds Lu <ref:2605.22809#pg1>.

Lu: Crucially, the model co-generates consistent LiDAR and demonstrates strong generalization to real world footage Tom.

Title and authors: Tom: It’s a scalable solution for data scarcity in safety critical validation and deployment of autonomous systems, which is really important stuff for the field Jane.

Meng: From an engineering side, I have to ask about the cost. Running this kind of diffusion model at scale with all those multi sensor outputs sounds like it would be computationally heavy Lu.

Lu: It is expensive because you’re running complex generative models, but they do show that view concatenation achieves better FID in the image only setting compared to channel concatenation Tom.

Tom: And the joint image LiDAR training shows that the full model achieves an LPIPS of zero point three one six, which is better than their channel concatenation variant at zero point three four six Jane.

Jane: That's a small number, but it shows they are getting good perceptual quality across both modalities when they train them together Lu.

Meng: I still worry about the temporal drift aspect. The paper mentions that temporal drift can happen over extended rollouts, which means long term sequences might lose coherence Tom.

Lu: They acknowledge that, and they suggest incorporating a more robust longer video generative backbone designed for long range consistency to address that Jane.

Tom: So the limitation is about very long sequences where things start to drift apart temporally Lu.

Jane: And they say a way to fix that would be expanding the auto-regressive conditioning window to look at older contexts, like t minus k, for better scene level consistency Lu.

Tom: So it’s not perfect yet on very long stretches, but they have a plan to make it more robust Jane.

Meng: I'd be interested in how this performs when we try to deploy it on edge hardware. Does the inference cost become too prohibitive for real time systems?

Lu: The paper focuses mostly on the generation quality and consistency metrics, but they mention that the separate processing streams for camera and LiDAR data allow the model to learn modality specific representations Tom.

Jane: That separation is important because it lets them train specialized parts while still fusing information through those shared attention layers Lu.

Tom: So, they’ve managed to keep the complexity manageable by keeping those streams separate but communicating effectively Jane.

Lu: And for the future, they are looking at making these models more efficient so they can actually run where we need them, not just in a lab setting Tom.

Jane: It sounds like a solid piece of work that tackles a huge problem in AV data scarcity by using three deeGS and diffusion models to create realistic sensor logs Lu <ref:2605.22809#pg1>.

The paper's summary: Tom: So we're talking about Sensor2Sensor today—it’s a new way to take those everyday videos from dashcams or the internet and turn them into actual sensor data logs that are useful for testing self-driving cars in the real world.

Jane: Right, so imagine you have a normal video of someone driving, and instead of just looking at the picture, you get all kinds of sensor information—like multiple camera angles and even LiDAR point clouds—for that exact same moment.

Lu: Essentially, this paper is about bridging that gap between what we see in a single video and what an actual car's sensor suite produces. It’s tackling the problem of needing massive amounts of diverse data for safety testing across different types of vehicles.

Tom: But the big hurdle is getting those paired datasets, right? You don't have perfect videos that already have LiDAR data attached for every single scenario.

Jane: That’s where they propose a synthetic data-pairing pipeline, and it uses something called dynamic three dimensional Gaussian Splatting or three deeGS to reconstruct the scene from existing AV logs.

Lu: So they take what we already have—real AV logs—and use that to build a four dimensional representation of the scene, which is really neat because it helps create something new.

Tom: And then from that reconstructed scene, they render synthetic dashcam views using camera parameters sampled from real world dashcams, which gives them the paired training data of synthetic dashcam videos and real AV logs.

Jane: That process is how they solve the lack of paired training data by creating this required corpus for validation.

Lu: The core of it is a conditional diffusion model that takes that single dashcam video as input and generates both eight different camera images and the LiDAR point clouds simultaneously.

Tom: And they make sure everything stays consistent using cross sensor attention for the image and three dimensional attention blocks for the multiple camera views.

Jane: The LiDAR generation uses a separate VAE that encodes depth, intensity, and elongation into a latent space where it minimizes losses for those components and also includes binary cross entropy loss for the validity mask.

Lu: They’re not just guessing shapes; they are explicitly training the model to generate plausible three dimensional geometry with features like range and intensity.

Tom: They use a total loss that includes one-to-one reconstruction losses for those things, plus LPIPS losses on surface normals and elongation to make sure the generated LiDAR looks perceptually good.

The paper's summary: Jane: They also introduced cross sensor attention within each U net block, which is designed to help the image features and the LiDAR features talk to each other continuously.

Lu: That cross sensor attention is key for making sure the camera view and the point cloud match up correctly, right?

Tom: Exactly. But they also extended this into auto-regressive generation using an algorithm called DAgger to help combat temporal inconsistency when converting third party videos into driving logs.

Jane: The DAgger algorithm augments the training context by including the model's own generations, which helps improve robustness by mitigating short term error accumulation.

Lu: So they’re not just generating one frame; they are doing this iterative process of generating rollouts and retraining on that new context to make the video sequence more stable.

Tom: It’s an interesting approach because it moves beyond just a single snapshot and tries to model how things evolve over time in a driving scene.

Jane: It means we can use tons of real world driving videos—even those from the internet or dashcams—to train and validate autonomous systems that are tailored to a specific vehicle type.

Lu: The authors show that this method achieves state of the art fidelity in cross embodiment sensor generation, which means it’s converting challenging in the wild videos into realistic data formats.

Tom: And they have some solid numbers supporting that, like Frechet Inception Distance for image realism and Frechet Video Distance for video realism.

Jane: Qualitatively, the paper shows that the generated LiDAR point clouds are cleaner than baselines and they maintain cross modal consistency between the image and LiDAR.

Lu: And human evaluations back this up too; it’s top preferred in about eighty-three percent of image cases for dashcam data, and over ninety four percent in pairwise comparisons against another method called X Drive.

Tom: So, to wrap up the main point—Sensor2Sensor is a novel generative paradigm that bridges the embodiment gap between consumer driving videos and the complex sensor suites needed for autonomous driving validation.

Jane: By using 4DGS based data pairing and a conditional diffusion architecture, they convert monocular third party videos into synchronized multiview camera streams and LiDAR point clouds <ref:2605.22809#pg1>.

Lu: Crucially, the model co generates consistent LiDAR and demonstrates strong generalization to real world footage.

The paper's summary: Tom: It’s a scalable solution for data scarcity in safety critical validation and deployment of autonomous systems, which is really important stuff for the field.

Meng: From an engineering side, I have to ask about the cost. Running this kind of diffusion model at scale with all those multi sensor outputs sounds like it would be computationally heavy.

Lu: It is expensive because you’re running complex generative models, but they do show that view concatenation achieves better FID in the image only setting compared to channel concatenation.

Tom: And the joint image LiDAR training shows that the full model achieves an LPIPS of zero point three one six, which is better than their channel concatenation variant at zero point three four six.

Jane: That's a small number, but it shows they are getting good perceptual quality across both modalities when they train them together.

Meng: I still worry about the temporal drift aspect. The paper mentions that temporal drift can happen over extended rollouts, which means long term sequences might lose coherence.

Lu: They acknowledge that, and they suggest incorporating a more robust longer video generative backbone designed for long range consistency to address that.

Tom: So the limitation is about very long sequences where things start to drift apart temporally.

Jane: And they say a way to fix that would be expanding the auto-regressive conditioning window to look at older contexts, like t minus k, for better scene level consistency.

Lu: So it’s not perfect yet on very long stretches, but they have a plan to make it more robust.

Meng: I'd be interested in how this performs when we try to deploy it on edge hardware. Does the inference cost become too prohibitive for real time systems?

Lu: The paper focuses mostly on the generation quality and consistency metrics, but they mention that the separate processing streams for camera and LiDAR data allow the model to learn modality specific representations.

Jane: That separation is important because it lets them train specialized parts while still fusing information through those shared attention layers.

Tom: So they’ve managed to keep the complexity manageable by keeping those streams separate but communicating effectively.

Lu: And for the future, they are looking at making these models more efficient so they can actually run where we need them, not just in a lab setting.

Jane: It sounds like a solid piece of work that tackles a huge problem in AV data scarcity by using three deeGS and diffusion models to create realistic sensor logs.

The paper's improvements: Tom: So we're talking about the improvements they suggest for Sensor2Sensor today—they aren't just stopping at what they did, right?

Jane: Right, so they’re looking at ways to make this system even better than just converting one video into sensor data.

Lu: They are focusing on fixing the temporal drift issue mentioned earlier, which happens when the model tries to generate really long sequences and things start losing their coherence over time.

Tom: So how do we fix that instability in the generated driving logs?

Jane: They suggest incorporating a more robust longer video generative backbone, something designed specifically to maintain consistency over a longer duration.

Lu: That would mean building a bigger engine for the diffusion model so it can handle those long-range temporal dependencies better.

Tom: A bigger engine sounds like more computational load, which is always a concern when you’re thinking about shipping this to something that actually runs in the field.

Jane: It is more work, but they also propose expanding the auto-regressive conditioning window to look at older context, like time minus some value k.

Lu: That way, the model gets to see a richer temporal context when it’s generating things now, which helps keep the scene level consistent.

Tom: So instead of just looking at what happened in the last few seconds, it looks further back to understand the whole situation better.

Jane: It means we get a more complete picture of what's happening in that driving scene across time, not just isolated moments.

Lu: That approach should help make the generated logs much more realistic for longer drives or complex maneuvers.

Tom: It’s about making sure the model understands the flow of time in a way that doesn't just drift apart over time.

Jane: The implication is that we could generate driving data that is usable for testing longer, more complicated autonomous driving scenarios.

Lu: If it works, it means we can train and test AI systems on much more realistic, long-term driving footage.

Tom: So they are tackling the scalability and stability of the generation process itself now.

Jane: And that leads us to thinking about how this improved generation quality actually helps us with autonomous vehicle safety validation.

Conclusion: Tom: So, to wrap up, Sensor2Sensor is this novel generative paradigm that bridges the embodiment gap between consumer driving videos and the complex sensor suites required for autonomous driving validation.

Jane: Exactly, by using 4DGS based data pairing and a conditional diffusion architecture, they convert monocular third party videos into synchronized multiview camera streams and LiDAR point clouds <ref:2605.22809#pg1>.

Lu: Crucially, the model co generates consistent LiDAR and demonstrates strong generalization to real world footage.

Tom: It’s a scalable solution for data scarcity in safety critical validation and deployment of autonomous systems, which is really important stuff for the field.

Jane: By unlocking large-scale driving videos for AV development, our approach provides a scalable solution to data scarcity for safety critical validation and deployment of safety critical autonomous systems.

Lalam: I’m sure of it; the paper shows they managed to get high fidelity across different sensor types, which is what AV companies desperately need right now.

Meng: I just worry about the sheer computational cost of running these massive diffusion models at scale for real time operations.

Lu: It is expensive because you’re running complex generative models, but they do show that view concatenation achieves better FID in the image only setting compared to channel concatenation.

Tom: And the joint image LiDAR training shows that the full model achieves an LPIPS of zero point three one six, which is better than their channel concatenation variant at zero point three four six.

Jane: That small number shows they are getting good perceptual quality across both modalities when they train them together, which is a big deal for realism.

Meng: I still worry about the temporal drift aspect. The paper mentions that temporal drift can happen over extended rollouts, which means long term sequences might lose coherence.

Lu: They acknowledge that, and they suggest incorporating a more robust longer video generative backbone designed for long range consistency to address that.

Tom: So the limitation is about very long sequences where things start to drift apart temporally.

Jane: And they say a way to fix that would be expanding the auto-regressive conditioning window to look at older contexts, like time minus some value k, for better scene level consistency.

Lu: That way, the model gets to see a richer temporal context when it’s generating things now, which helps keep the scene level consistent.

Meng: If it works, it means we can train and test AI systems on much more realistic long term driving footage.

Tom: It’s about making sure the model understands the flow of time in a way that doesn't just drift apart over time.

Jane: The implication is that we could generate driving data that is usable for testing longer, more complicated autonomous driving scenarios.

Lu: If it works, it means we can train and test AI systems on much more realistic, long term driving footage.

Tom: So they are tackling the scalability and stability of the generation process itself now.

Jane: And that leads us to thinking about how this improved generation quality actually helps us with autonomous vehicle safety validation.

Jiahao Wang, Bo Sun, Yijing Bai, Vincent Casser, Songyou Peng, Zehao Zhu, Meng-Li Shih, Xander Masotto, Shih-Yang Su, Kanaad Parvate,  Tiancheng Ge–Ge1,  Linn Bieske–Ge1, ◗Dragomir Anguelov◗Mingxing Tan◗Chiyu Max Jiang, Johns Hopkins University, Google DeepMind, University of Washington

Waymo · Google DeepMind · University of Washington

cs.CV

Submitted: 2026-05-21

Updated: 2026-05-22

Comments: Accepted by CVPR 2026

Journal ref: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2026, pp. 32093-32102

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 100/100

The gist: Sensor2Sensor is a novel generative paradigm that translates in-the-wild monocular dashcam videos into a high-fidelity…

Key concepts

Sensor2Sensor
A generative system that takes a single video from a dashcam and creates a complete set of sensor data—including several camera views and LiDAR point clouds. This bridges the gap between consumer videos and the complex, multi-modal sensor requirements for autonomous vehicle testing.
Synthetic Data-Pairing Pipeline
A method to create training examples when real paired data is scarce. It uses 3D Gaussian Splatting (4DGS) to reconstruct scenes from existing AV logs, then renders synthetic dashcam views with realistic parameters. This generates the necessary pairs of (synthetic video, real log) for model training.
Conditional Diffusion Model
The core generative architecture that creates the output. It is a diffusion model designed to generate both images and LiDAR data simultaneously, conditioned on the input video. It uses cross-sensor attention to ensure the generated camera views and LiDAR point clouds are consistent with each other.

Terminology

Summary

Sensor2Sensor is a novel generative paradigm that translates in-the-wild monocular dashcam videos into a high-fidelity, multi-modal sensor suite (AV logs) comprising multi-view camera images and LiDAR point clouds <ref:2605. p. 022809#pg2>. This enables cross-embodiment sensor conversion to maximally leverage real-world long-tail data for AV system validation <ref:2605. p. 022809#pg2>.

Data Bridging Strategy

The paper addresses the lack of paired training data by proposing a novel synthetic data-pairing pipeline <ref:2605.22809#pg2>. This pipeline leverages existing AV logs to reconstruct 4D scene representations via dynamic 3D Gaussian Splatting (3DGS) <ref:2605. p. 122809#pg2>. From this reconstructed scene, synthetic dashcam views are rendered with augmented intrinsic and extrinsic parameters sampled from real-world dashcam distributions <ref:2605. p. 122809#pg2>. This process yields the required paired training corpus: (synthetic dashcam, real AV log) <ref:2605. p. 122809#pg2>.

Generative Model Architecture

Sensor2Sensor utilizes a conditional diffusion model designed for multi-sensor (eight cameras) and multi-modal (camera and LiDAR) output, conditioned on the input dashcam video <ref:2605.22809#pg2>. The architecture simultaneously generates multi-view images C = 8i=1 and LiDAR point clouds L <ref:2605. p. 322809#pg2>. Multi-sensor consistency is enforced via cross-sensor attention, and multi-view consistency is maintained with 3D attention blocks <ref:2605. p. 322809#pg2>.

Multi-Modal Generation Details

The model employs separate VAEs for image and LiDAR generation, where the LiDAR VAE jointly encodes depth, intensity, and elongation <ref:2605.22809#pg2>. The LiDAR generation uses a VAE architecture that minimizes a total loss including L1 reconstruction losses for range, elongation, and intensity <ref:2605. p. 322809#pg2>. Cross-sensor attention is introduced within each U-Net block to promote continuous information interchange between image features f iC and LiDAR features f iL <ref:2605. p. 322809#pg2>.

Temporal Consistency Enhancement

To convert third-party videos into driving logs, the model extends its capabilities for auto-regressive generation <ref:2605.22809#pg2>. The DAgger algorithm is introduced to augment the training context with the model’s own generations to combat temporal inconsistency <ref:2605. p. 422809#pg2>. This iterative process involves generating rollout videos and training a new model on the resulting context, which improves robustness <ref:2605. p. 422809#pg2>.

Evaluation and Generalization

The method is validated using metrics such as Frechet Inception Distance (FID) for image realism and Frechet Video Distance (FVD) for video realism <ref:2605.22809#pg2>. Qualitative results show that Sensor2Sensor achieves state-of-the-art fidelity, converting challenging in-the-wild videos into realistic and usable data formats <ref:2605. p. 122809#pg2>. Furthermore, the model demonstrates strong qualitative generalization to diverse and challenging in-the-wild driving footage <ref:2605. p. 722809#pg2>.

Conclusion

Sensor2Sensor is a novel generative paradigm that bridges the embodiment gap between consumer driving videos and the complex, multi-modal sensor suites required for AV validation <ref:2605.22809#pg2>. Leveraging a 4DGS-based data pairing pipeline and a conditional diffusion architecture, Sensor2Sensor converts monocular third-party videos into synchronized multiview camera streams and LiDAR point clouds, achieving state-of-the-art performance in cross-embodiment sensor generation <ref:2605. p. 7 p. 722809#pg2>. Crucially, the model co-generates consistent LiDAR and demonstrates strong generalization to real-world footage <ref:2605. p. 7 p. 722809#pg2>. By unlocking large-scale driving videos for AV development, our approach provides a scalable solution to data scarcity for safety-critical validation and deployment of safety-critical autonomous systems <ref:2605. p. 7 p. 722809#pg2>.

Supplementary Material Details

The training pipeline involves four steps, starting with base conditioning single frame generation <ref:2605.22809#pg2>. Step 3 utilizes DAgger Data Generation to create a degraded dataset for augmentation <ref:2605.22809#pg2>. The model architecture features separate processing streams for camera and LiDAR data, allowing the model to learn modality-specific representations while fusing information through shared attention layers <ref:2605. p. 322809#pg2>. The DAgger finetuning strategy mitigates short-term error accumulation, though temporal drift remains a known challenge for long-horizon sequences <ref:2605.22809#pg2>.

Limitations and Future Work

A limitation is that temporal drift can occur over extended rollouts, leading to a gradual loss of long-range temporal coherence <ref:2605.22809#pg2>. This limitation could be addressed by incorporating a more robust longer video generative backbone designed for long-range consistency <ref:2605.22809#pg2>. A complementary solution would be to expand the auto-regressive conditioning window to attend to a richer temporal context, such as t − k, for better scene-level consistency <ref:2605. p. 422809#pg2>.

Synthetic Camera Synthesis

One important component of the pipeline is the utilization of 4D Gaussian Splatting (4DGS) to synthesize paired training data by simulating third-party camera views <ref:2605.22809#pg2>. The diffusion model is trained to map these synthetic inputs (Isynth), which may contain minor reconstruction artifacts, to pristine, ground-truth real sensor data (Oreal) <ref:2605. p. 222809#pg2>. This training objective forces the network to learn robust spatial and semantic mappings between the monocular view and the target sensor suite <ref:2605. p. 022809#pg2>.

Dashcam Parameter Distribution

Camera parameters are sampled via a two-stage process, first selecting a vehicle category, then sampling 6-DoF poses from category-specific distributions <ref:2605.22809#pg2>. Intrinsics are drawn from calibrated real-world dashcams and augmented with uniform noise <ref:2605.22809#pg2>. Final outputs undergo exposure compensation and gamma correction for lighting normalization <ref:2605.22809#pg2>.

LiDAR VAE Training

The LiDAR VAE is trained to jointly encode depth, intensity, and elongation into a compact latent space <ref:2605.22809#pg2>. The total loss for the LiDAR VAE includes L1 reconstruction losses for signal components and Binary Cross-Entropy Loss for the validity mask <ref:2605. p. 322809#pg2>. Additionally, LPIPS losses are applied to surface normals, elongation, intensity, and validity to ensure perceptual quality <ref:2605. p. 322809#pg2>.

Ablation Study Findings

The ablation study confirms that view concatenation (VC) achieves better FID in the image-only setting compared to channel concatenation (CC) <ref:2605.22809#pg2>. Joint image-LiDAR training shows that the full model achieves LPIPS 0. p. 7316, outperforming the CC variant (0. p. 7346) while remaining competitive with image-only VC (0. p. 7307) <ref:2605. p. 022809#pg2>. DAgger finetuning improves video quality metrics like FVD and FID <ref:2605. p. 722809#pg2>.

Downstream Tasks

Perception models trained on real data applied directly to the generated data show comparable performance in LiDAR detection and image segmentation <ref:2605.22809#pg2>. These results confirm the fidelity of the generation, indicating strong alignment with real-world distributions <ref:2605. p. 722809#pg2>.

Human Evaluation Insights

Human evaluation shows that Sensor2Sensor is top-preferred in 83.46% of image cases and 68. p. 608% for LiDAR on dashcam data <ref:2605. p. 022809#pg2>. For internet data, it is top-preferred in 84. p. 662% of image cases and 58. p. 646% for LiDAR <ref:2605. p. 022809#pg2>. Pairwise comparisons show Sensor2Sensor is preferred over X-Drive in over 94% of image cases and 85% for LiDAR <ref:2605. p. 622809#pg2>.

LiDAR Generation Fidelity

Qualitatively, the model shows a superior ability to reconstruct plausible 3D geometry for both nearby actors and the static environment <ref:2605.22809#pg2>. The generated LiDAR points are cleaner, with fewer noise artifacts and more accurate intensity rendering compared to baselines <ref:2605. p. 522809#pg2>. Sensor2Sensor achieves cross-modal consistency between image and LiDAR, faithfully generating safety-critical objects <ref:2605. p. 622809#pg2>.

Image-LiDAR Alignment

Visualizations confirm that the generated LiDAR point cloud accurately reflects the geometric details observed in the synthesized image view <ref:2605.22809#pg2>. This demonstrates precise spatial registration between the two modalities <ref:2605.22809#pg2>. The model learns a coherent underlying 3D representation of the scene <ref:2605. p. 522809#pg2>.

Final Summary

Sensor2Sensor is a novel generative paradigm that bridges the embodiment gap between consumer driving videos and the complex, multi-modal sensor suites required for AV validation <ref:2605.22809#pg2>. Leveraging a 4DGS-based data pairing pipeline and a conditional diffusion architecture, Sensor2Sensor converts monocular third-party videos into synchronized multiview camera streams and LiDAR point clouds, achieving state-of-the-art performance in cross-embodiment sensor generation <ref:2605.22809#pg2>. Crucially, the model co-generates consistent LiDAR and demonstrates strong generalization to real-world footage <ref:2605.22809#pg2>. By unlocking large-scale driving videos for AV development, our approach provides a scalable solution to data scarcity for safety-critical validation and deployment of safety-critical autonomous systems <ref:2605.22809#pg2>.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Cross-Embodiment Sensor Conversion for Autonomous Driving: The system can translate in-the-wild monocular dashcam videos into high-fidelity, multi-modal sensor suites comprising multi-view camera images and LiDAR point clouds specific to a target vehicle embodiment. This enables cross-embodiment sensor conversion to maximally leverage real-world longtail data for AV system validation.

  2. Synthetic Data Pairing Pipeline: The pipeline uses dynamic 3D Gaussian Splatting (4DGS) to reconstruct scenes from existing AV logs, which are then rendered into novel, synthetic dashcam views with augmented intrinsic and extrinsic parameters. This process yields the required paired training corpus of (synthetic dashcam, real AV log), circumventing the lack of paired training data.

  3. Multi-modal Diffusion Model: The system utilizes a conditional diffusion model that simultaneously generates multi-view images and LiDAR point clouds conditioned on a single camera input. It enforces multi-sensor consistency via cross-sensor attention and multi-view consistency using 3D attention blocks to produce synchronized, geometrically accurate sensor logs.

  4. Auto-regressive Video Generation: By introducing the DAgger algorithm, the system extends auto-regressive generation to model the conditional probability distribution of multi-view images and LiDAR point clouds over time. This allows for the conversion of third-party videos into driving logs, maintaining temporal consistency by training on generated rollouts to mitigate drifting.

  5. Generalization to In-the-Wild Footage: The system demonstrates strong qualitative generalization when applied to diverse uncurated videos from internet, dashcams, and other third-party sources. It converts monocular inputs into coherent multi-sensor AV logs while preserving critical scene elements in challenging, unseen environments such as night-time near collisions or active incidents.

Sources

Related papers