Sensor2Sensor: Cross-Embodiment Sensor Conversion for Autonomous Driving
summary
The gist
Sensor2Sensor is a novel generative paradigm that translates in-the-wild monocular dashcam videos into a high-fidelity…
In short
Sensor2Sensor translates single dashcam videos into complete sensor suites (multiple camera views and LiDAR) needed for autonomous vehicle validation. It uses a novel synthetic data pairing pipeline based on 3D Gaussian Splatting to create training data where none existed. The resulting generative model produces consistent, high-fidelity multi-modal logs, allowing real-world driving footage to be used for safety testing even when paired sensor data is missing.
Key concepts
- Sensor2Sensor
- A generative system that takes a single video from a dashcam and creates a complete set of sensor data—including several camera views and LiDAR point clouds. This bridges the gap between consumer videos and the complex, multi-modal sensor requirements for autonomous vehicle testing.
- Synthetic Data-Pairing Pipeline
- A method to create training examples when real paired data is scarce. It uses 3D Gaussian Splatting (4DGS) to reconstruct scenes from existing AV logs, then renders synthetic dashcam views with realistic parameters. This generates the necessary pairs of (synthetic video, real log) for model training.
- Conditional Diffusion Model
- The core generative architecture that creates the output. It is a diffusion model designed to generate both images and LiDAR data simultaneously, conditioned on the input video. It uses cross-sensor attention to ensure the generated camera views and LiDAR point clouds are consistent with each other.
Terminology used across episodes
This episode discusses
- Sensor2Sensor: Cross-Embodiment Sensor Conversion for Autonomous Driving · Paper Radio
- PAGS: Priority-Adaptive Gaussian Splatting for Dynamic Driving Scenes
- Cosmos World Foundation Model Platform for Physical AI
- V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning
- Text2Scenario: Text-Driven Scenario Generation for Autonomous Driving Test
- Driv3R: Learning Dense 4D Reconstruction for Autonomous Driving
- Genesis: Multimodal Driving Scene Generation with Spatio-Temporal and Cross-Modal Consistency
- World Models
- Training Agents Inside of Scalable World Models
- GAIA-1: A Generative World Model for Autonomous Driving
- S cubed Gaussian: Self-Supervised Street Gaussians for Autonomous Driving
- Txt2Sce: Scenario Generation for Autonomous Driving System Testing Based on Textual Reports
- Cosmos Policy: Fine-Tuning Video Models for Visuomotor Control and Planning
- GenEx: Generating an Explorable World
- From Dashcam Videos to Driving Simulations: Stress Testing Automated Vehicles against Rare Events
- Cosmos-Drive-Dreams: Scalable Synthetic Driving Data Generation with World Foundation Models
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- GenMM: Geometrically and Temporally Consistent Multimodal Data Generation for Video and LiDAR
- Wan: Open and Advanced Large-Scale Video Generative Models
- Flux4D: Flow-based Unsupervised 4D Reconstruction
- EvoWorld: Evolving Panoramic World Generation with Explicit 3D Memory · Paper Radio
The paper
Sensor2Sensor: Cross-Embodiment Sensor Conversion for Autonomous Driving · Read on arXiv
Jiahao Wang, Bo Sun, Yijing Bai, Vincent Casser, Songyou Peng, Zehao Zhu, Meng-Li Shih, Xander Masotto, Shih-Yang Su, Kanaad Parvate, Tiancheng Ge–Ge1, Linn Bieske–Ge1, ◗Dragomir Anguelov◗Mingxing Tan◗Chiyu Max Jiang, Johns Hopkins University, Google DeepMind, University of Washington
Waymo · Google DeepMind · University of Washington
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: I'm Tom.
Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.
Tom: We're looking at the paper: "Sensor2Sensor: Cross-Embodiment Sensor Conversion for Autonomous Driving".
Jane: Here's the gist of the paper. Sensor2Sensor is a novel generative paradigm that translates in-the-wild monocular dashcam videos into a high-fidelity, multi-modal sensor suite (AV logs) comprising multi-view camera images and LiDAR point clouds <ref:2605.22809#pg2>.
Tom: This enables cross-embodiment sensor conversion to maximally leverage real-world long-tail data for AV system validation <ref:2605.22809#pg2>.
Jane: Data Bridging Strategy The paper addresses the lack of paired training data by proposing a novel synthetic data-pairing pipeline <ref:2605.22809#pg2>.
Tom: This pipeline leverages existing AV logs to reconstruct 4D scene representations via dynamic 3D Gaussian Splatting (3DGS) <ref:2605.22809#pg2>.
Title and authors: Tom: So we're talking about Sensor2Sensor today. It’s a new way to take those everyday videos from dashcams or the internet and turn them into actual sensor data logs that are useful for testing self-driving cars in the real world Jane.
Jane: Right, so imagine you have a normal video of someone driving, and instead of just looking at the picture, you get all kinds of sensor information—like multiple camera angles and even LiDAR point clouds—for that exact same moment Tom.
Lu: Essentially, this paper is about bridging that gap between what we see in a single video and what an actual car's sensor suite produces Jane. It’s tackling the problem of needing massive amounts of diverse data for safety testing across different types of vehicles Tom.
Meng: But the big hurdle is getting those paired datasets, right? You don't have perfect videos that already have LiDAR data attached for every single scenario Jane.
Lu: That’s where they propose a synthetic data-pairing pipeline, and it uses something called dynamic three dimensional Gaussian Splatting or three deeGS to reconstruct the scene from existing AV logs Tom <ref:2605.22809#pg1>.
Tom: So they take what we already have—real AV logs—and use that to build a 4D representation of the scene, which is really neat because it helps create something new Jane <ref:2605.22809#pg1>.
Jane: And then from that reconstructed scene, they render synthetic dashcam views using camera parameters sampled from real-world dashcams, which gives them the paired training data of synthetic dashcam videos and real AV logs Lu.
Lu: That process is how they solve the lack of paired training data by creating this required corpus for validation Tom.
Tom: The core of it is a conditional diffusion model that takes that single dashcam video as input and generates both eight different camera images and the LiDAR point clouds simultaneously Jane.
Jane: And they make sure everything stays consistent using cross sensor attention for the image and three dee attention blocks for the multiple camera views Lu <ref:2605.22809#pg1>.
Lu: The LiDAR generation uses a separate VAE that encodes depth, intensity, and elongation into a latent space where it minimizes losses for those components and also includes binary cross entropy loss for the validity mask Tom.
Tom: So they’re not just guessing shapes; they are explicitly training the model to generate plausible three dee geometry with features like range and intensity Jane <ref:2605.22809#pg1>.
Jane: They use a total loss that includes L1 reconstruction losses for those things, plus LPIPS losses on surface normals and elongation to make sure the generated LiDAR looks perceptually good Lu.
Title and authors: Lu: They also introduced cross sensor attention within each U-Net block, which is designed to help the image features and the LiDAR features talk to each other continuously Tom.
Tom: That cross sensor attention is key for making sure the camera view and the point cloud match up correctly, right?
Jane: Exactly. But they also extended this into auto-regressive generation using an algorithm called DAgger to help combat temporal inconsistency when converting third-party videos into driving logs Lu.
Lu: The DAgger algorithm augments the training context by including the model's own generations, which helps improve robustness by mitigating short term error accumulation Tom.
Tom: So they’re not just generating one frame; they are doing this iterative process of generating rollouts and retraining on that new context to make the video sequence more stable Jane.
Jane: It’s an interesting approach because it moves beyond just a single snapshot and tries to model how things evolve over time in a driving scene Lu.
Tom: So, what does this mean for the people who actually use these models? Does it help us get better autonomous systems?
Jane: It means we can use tons of real world driving videos—even those from the internet or dashcams—to train and validate autonomous systems that are tailored to a specific vehicle type Lu.
Lu: The authors show that this method achieves state-of-the-art fidelity in cross embodiment sensor generation, which means it’s converting challenging in-the-wild videos into realistic data formats Tom.
Tom: And they have some solid numbers supporting that, like Frechet Inception Distance for image realism and Frechet Video Distance for video realism Jane.
Jane: Qualitatively, the paper shows that the generated LiDAR point clouds are cleaner than baselines and they maintain cross modal consistency between the image and LiDAR Lu.
Lu: And human evaluations back this up too; it’s top preferred in about eighty-three percent of image cases for dashcam data, and over ninety-four percent in pairwise comparisons against another method called X Drive Tom.
Tom: So, to wrap up the main point—Sensor2Sensor is a novel generative paradigm that bridges the embodiment gap between consumer driving videos and the complex sensor suites needed for autonomous driving validation Jane.
Jane: By using 4DGS based data pairing and a conditional diffusion architecture, they convert monocular third-party videos into synchronized multiview camera streams and LiDAR point clouds Lu <ref:2605.22809#pg1>.
Lu: Crucially, the model co-generates consistent LiDAR and demonstrates strong generalization to real world footage Tom.
Title and authors: Tom: It’s a scalable solution for data scarcity in safety critical validation and deployment of autonomous systems, which is really important stuff for the field Jane.
Meng: From an engineering side, I have to ask about the cost. Running this kind of diffusion model at scale with all those multi sensor outputs sounds like it would be computationally heavy Lu.
Lu: It is expensive because you’re running complex generative models, but they do show that view concatenation achieves better FID in the image only setting compared to channel concatenation Tom.
Tom: And the joint image LiDAR training shows that the full model achieves an LPIPS of zero point three one six, which is better than their channel concatenation variant at zero point three four six Jane.
Jane: That's a small number, but it shows they are getting good perceptual quality across both modalities when they train them together Lu.
Meng: I still worry about the temporal drift aspect. The paper mentions that temporal drift can happen over extended rollouts, which means long term sequences might lose coherence Tom.
Lu: They acknowledge that, and they suggest incorporating a more robust longer video generative backbone designed for long range consistency to address that Jane.
Tom: So the limitation is about very long sequences where things start to drift apart temporally Lu.
Jane: And they say a way to fix that would be expanding the auto-regressive conditioning window to look at older contexts, like t minus k, for better scene level consistency Lu.
Tom: So it’s not perfect yet on very long stretches, but they have a plan to make it more robust Jane.
Meng: I'd be interested in how this performs when we try to deploy it on edge hardware. Does the inference cost become too prohibitive for real time systems?
Lu: The paper focuses mostly on the generation quality and consistency metrics, but they mention that the separate processing streams for camera and LiDAR data allow the model to learn modality specific representations Tom.
Jane: That separation is important because it lets them train specialized parts while still fusing information through those shared attention layers Lu.
Tom: So, they’ve managed to keep the complexity manageable by keeping those streams separate but communicating effectively Jane.
Lu: And for the future, they are looking at making these models more efficient so they can actually run where we need them, not just in a lab setting Tom.
Jane: It sounds like a solid piece of work that tackles a huge problem in AV data scarcity by using three deeGS and diffusion models to create realistic sensor logs Lu <ref:2605.22809#pg1>.
The paper's summary: Tom: So we're talking about Sensor2Sensor today—it’s a new way to take those everyday videos from dashcams or the internet and turn them into actual sensor data logs that are useful for testing self-driving cars in the real world.
Jane: Right, so imagine you have a normal video of someone driving, and instead of just looking at the picture, you get all kinds of sensor information—like multiple camera angles and even LiDAR point clouds—for that exact same moment.
Lu: Essentially, this paper is about bridging that gap between what we see in a single video and what an actual car's sensor suite produces. It’s tackling the problem of needing massive amounts of diverse data for safety testing across different types of vehicles.
Tom: But the big hurdle is getting those paired datasets, right? You don't have perfect videos that already have LiDAR data attached for every single scenario.
Jane: That’s where they propose a synthetic data-pairing pipeline, and it uses something called dynamic three dimensional Gaussian Splatting or three deeGS to reconstruct the scene from existing AV logs.
Lu: So they take what we already have—real AV logs—and use that to build a four dimensional representation of the scene, which is really neat because it helps create something new.
Tom: And then from that reconstructed scene, they render synthetic dashcam views using camera parameters sampled from real world dashcams, which gives them the paired training data of synthetic dashcam videos and real AV logs.
Jane: That process is how they solve the lack of paired training data by creating this required corpus for validation.
Lu: The core of it is a conditional diffusion model that takes that single dashcam video as input and generates both eight different camera images and the LiDAR point clouds simultaneously.
Tom: And they make sure everything stays consistent using cross sensor attention for the image and three dimensional attention blocks for the multiple camera views.
Jane: The LiDAR generation uses a separate VAE that encodes depth, intensity, and elongation into a latent space where it minimizes losses for those components and also includes binary cross entropy loss for the validity mask.
Lu: They’re not just guessing shapes; they are explicitly training the model to generate plausible three dimensional geometry with features like range and intensity.
Tom: They use a total loss that includes one-to-one reconstruction losses for those things, plus LPIPS losses on surface normals and elongation to make sure the generated LiDAR looks perceptually good.
The paper's summary: Jane: They also introduced cross sensor attention within each U net block, which is designed to help the image features and the LiDAR features talk to each other continuously.
Lu: That cross sensor attention is key for making sure the camera view and the point cloud match up correctly, right?
Tom: Exactly. But they also extended this into auto-regressive generation using an algorithm called DAgger to help combat temporal inconsistency when converting third party videos into driving logs.
Jane: The DAgger algorithm augments the training context by including the model's own generations, which helps improve robustness by mitigating short term error accumulation.
Lu: So they’re not just generating one frame; they are doing this iterative process of generating rollouts and retraining on that new context to make the video sequence more stable.
Tom: It’s an interesting approach because it moves beyond just a single snapshot and tries to model how things evolve over time in a driving scene.
Jane: It means we can use tons of real world driving videos—even those from the internet or dashcams—to train and validate autonomous systems that are tailored to a specific vehicle type.
Lu: The authors show that this method achieves state of the art fidelity in cross embodiment sensor generation, which means it’s converting challenging in the wild videos into realistic data formats.
Tom: And they have some solid numbers supporting that, like Frechet Inception Distance for image realism and Frechet Video Distance for video realism.
Jane: Qualitatively, the paper shows that the generated LiDAR point clouds are cleaner than baselines and they maintain cross modal consistency between the image and LiDAR.
Lu: And human evaluations back this up too; it’s top preferred in about eighty-three percent of image cases for dashcam data, and over ninety four percent in pairwise comparisons against another method called X Drive.
Tom: So, to wrap up the main point—Sensor2Sensor is a novel generative paradigm that bridges the embodiment gap between consumer driving videos and the complex sensor suites needed for autonomous driving validation.
Jane: By using 4DGS based data pairing and a conditional diffusion architecture, they convert monocular third party videos into synchronized multiview camera streams and LiDAR point clouds <ref:2605.22809#pg1>.
Lu: Crucially, the model co generates consistent LiDAR and demonstrates strong generalization to real world footage.
The paper's summary: Tom: It’s a scalable solution for data scarcity in safety critical validation and deployment of autonomous systems, which is really important stuff for the field.
Meng: From an engineering side, I have to ask about the cost. Running this kind of diffusion model at scale with all those multi sensor outputs sounds like it would be computationally heavy.
Lu: It is expensive because you’re running complex generative models, but they do show that view concatenation achieves better FID in the image only setting compared to channel concatenation.
Tom: And the joint image LiDAR training shows that the full model achieves an LPIPS of zero point three one six, which is better than their channel concatenation variant at zero point three four six.
Jane: That's a small number, but it shows they are getting good perceptual quality across both modalities when they train them together.
Meng: I still worry about the temporal drift aspect. The paper mentions that temporal drift can happen over extended rollouts, which means long term sequences might lose coherence.
Lu: They acknowledge that, and they suggest incorporating a more robust longer video generative backbone designed for long range consistency to address that.
Tom: So the limitation is about very long sequences where things start to drift apart temporally.
Jane: And they say a way to fix that would be expanding the auto-regressive conditioning window to look at older contexts, like t minus k, for better scene level consistency.
Lu: So it’s not perfect yet on very long stretches, but they have a plan to make it more robust.
Meng: I'd be interested in how this performs when we try to deploy it on edge hardware. Does the inference cost become too prohibitive for real time systems?
Lu: The paper focuses mostly on the generation quality and consistency metrics, but they mention that the separate processing streams for camera and LiDAR data allow the model to learn modality specific representations.
Jane: That separation is important because it lets them train specialized parts while still fusing information through those shared attention layers.
Tom: So they’ve managed to keep the complexity manageable by keeping those streams separate but communicating effectively.
Lu: And for the future, they are looking at making these models more efficient so they can actually run where we need them, not just in a lab setting.
Jane: It sounds like a solid piece of work that tackles a huge problem in AV data scarcity by using three deeGS and diffusion models to create realistic sensor logs.
The paper's improvements: Tom: So we're talking about the improvements they suggest for Sensor2Sensor today—they aren't just stopping at what they did, right?
Jane: Right, so they’re looking at ways to make this system even better than just converting one video into sensor data.
Lu: They are focusing on fixing the temporal drift issue mentioned earlier, which happens when the model tries to generate really long sequences and things start losing their coherence over time.
Tom: So how do we fix that instability in the generated driving logs?
Jane: They suggest incorporating a more robust longer video generative backbone, something designed specifically to maintain consistency over a longer duration.
Lu: That would mean building a bigger engine for the diffusion model so it can handle those long-range temporal dependencies better.
Tom: A bigger engine sounds like more computational load, which is always a concern when you’re thinking about shipping this to something that actually runs in the field.
Jane: It is more work, but they also propose expanding the auto-regressive conditioning window to look at older context, like time minus some value k.
Lu: That way, the model gets to see a richer temporal context when it’s generating things now, which helps keep the scene level consistent.
Tom: So instead of just looking at what happened in the last few seconds, it looks further back to understand the whole situation better.
Jane: It means we get a more complete picture of what's happening in that driving scene across time, not just isolated moments.
Lu: That approach should help make the generated logs much more realistic for longer drives or complex maneuvers.
Tom: It’s about making sure the model understands the flow of time in a way that doesn't just drift apart over time.
Jane: The implication is that we could generate driving data that is usable for testing longer, more complicated autonomous driving scenarios.
Lu: If it works, it means we can train and test AI systems on much more realistic, long-term driving footage.
Tom: So they are tackling the scalability and stability of the generation process itself now.
Jane: And that leads us to thinking about how this improved generation quality actually helps us with autonomous vehicle safety validation.
Conclusion: Tom: So, to wrap up, Sensor2Sensor is this novel generative paradigm that bridges the embodiment gap between consumer driving videos and the complex sensor suites required for autonomous driving validation.
Jane: Exactly, by using 4DGS based data pairing and a conditional diffusion architecture, they convert monocular third party videos into synchronized multiview camera streams and LiDAR point clouds <ref:2605.22809#pg1>.
Lu: Crucially, the model co generates consistent LiDAR and demonstrates strong generalization to real world footage.
Tom: It’s a scalable solution for data scarcity in safety critical validation and deployment of autonomous systems, which is really important stuff for the field.
Jane: By unlocking large-scale driving videos for AV development, our approach provides a scalable solution to data scarcity for safety critical validation and deployment of safety critical autonomous systems.
Lalam: I’m sure of it; the paper shows they managed to get high fidelity across different sensor types, which is what AV companies desperately need right now.
Meng: I just worry about the sheer computational cost of running these massive diffusion models at scale for real time operations.
Lu: It is expensive because you’re running complex generative models, but they do show that view concatenation achieves better FID in the image only setting compared to channel concatenation.
Tom: And the joint image LiDAR training shows that the full model achieves an LPIPS of zero point three one six, which is better than their channel concatenation variant at zero point three four six.
Jane: That small number shows they are getting good perceptual quality across both modalities when they train them together, which is a big deal for realism.
Meng: I still worry about the temporal drift aspect. The paper mentions that temporal drift can happen over extended rollouts, which means long term sequences might lose coherence.
Lu: They acknowledge that, and they suggest incorporating a more robust longer video generative backbone designed for long range consistency to address that.
Tom: So the limitation is about very long sequences where things start to drift apart temporally.
Jane: And they say a way to fix that would be expanding the auto-regressive conditioning window to look at older contexts, like time minus some value k, for better scene level consistency.
Lu: That way, the model gets to see a richer temporal context when it’s generating things now, which helps keep the scene level consistent.
Meng: If it works, it means we can train and test AI systems on much more realistic long term driving footage.
Tom: It’s about making sure the model understands the flow of time in a way that doesn't just drift apart over time.
Jane: The implication is that we could generate driving data that is usable for testing longer, more complicated autonomous driving scenarios.
Lu: If it works, it means we can train and test AI systems on much more realistic, long term driving footage.
Tom: So they are tackling the scalability and stability of the generation process itself now.
Jane: And that leads us to thinking about how this improved generation quality actually helps us with autonomous vehicle safety validation.
More episodes
- 2606.27866-FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- 2607.04680-A Physics-Regulated Neural Framework for Learning 3D Grain Growth Dynamics
- 2607.04919-When Do Foundation Models Pay Off? A Break-Even Analysis of Pretrained Time Series Forecasters
- 2607.05025-Uncertainty-aware damage identification in short-span bridges via physics-informed variational autoencoder
- 2607.12304-What Does a Temporal Benchmark Score Measure? Decomposing Channel Use in Video VLM Evaluation
- 2608.02877-Graph-of-Thoughts for LLM Causal Discovery with a Fairness-Pathway Case Study
- 2110.08294-Coherence boosting: When your pretrained language model is not paying enough attention
- 2210.01293-ThinkSum: Probabilistic reasoning over sets using large language models
- 2504.15270-Quicksviewer: An LMM with Early Video Compression using Annealed Gumbel-Softmax
- 2508.10683-Neural Machine Translation for Coptic-French: Strategies for Low-Resource Ancient Languages