MITO: A Millimeter-Wave Dataset and Simulator for Non-Line-of-Sight Perception

summary

The gist

The ability to observe the world is fundamental to reasoning and making informed decisions on how to interact with the environment MITO…

In short

MITO is a novel dataset of millimeter-wave images for everyday objects created using a robotic arm and radars. It includes synthetic images generated by a simulator that models real-world reflections, enabling perception in non-line-of-sight scenarios. The work establishes baselines for object segmentation and classification using these mmWave techniques.

Key concepts

MITO Dataset
This is a large collection of millimeter-wave (mmWave) images of 76 diverse everyday objects. It was captured using a UR5 robot with two different mmWave radars and an RGB-D camera, providing high-resolution data for studying non-line-of-sight perception.
Simulation Framework
An open-source simulator is developed to create synthetic mmWave images from any 3D shape. It models material properties by simulating reflections based on distance and surface normals, including variations for specular and edge reflections.
Non-Line-of-Sight (NLOS) Perception
This refers to the ability of a system to perceive objects even when they are not directly visible. The MITO framework is designed to improve this capability by using multi-spectral fusion and advanced simulation techniques for object segmentation and classification.
Multi-spectral Fusion
This technique combines information from two different mmWave frequencies (77 GHz and 24 GHz) during prompt point selection for object segmentation. This helps the system better filter out noise, improving accuracy in identifying objects when they are partially occluded.

Terminology used across episodes

This episode discusses

The paper

MITO: A Millimeter-Wave Dataset and Simulator for Non-Line-of-Sight Perception · Read on arXiv

Massachusetts Institute of Technology · Adobe Research

The ability to observe the world is fundamental to reasoning and making informed decisions on how to interact with the environment. However, optical perception can often be disrupted due to common occurrences like occlusions, which can pose challenges to existing vision systems. We present MITO, the first millimeter-wave (mmWave) dataset of diverse, everyday objects, collected using a UR5 robotic arm with two mmWave radars operating at different frequencies and an RGB-D camera. Unlike visible light, mmWave signals can penetrate common occlusions (e.g., cardboard boxes, fabric, plastic). MITO captures over 24 million mmWave frames and uses them to generate 550 high-resolution mmWave images in line-of-sight and non-light-of-sight (NLOS), as well as RGB-D images, segmentation masks, and raw mmWave signals, taken from 76 different objects. We develop an open-source simulation tool that can be used to generate synthetic mmWave images for any 3D triangle mesh. Finally, we establish benchmarks for NLOS segmentation and classification to demonstrate the utility of our dataset and simulator for enabling broader NLOS perception.

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "MITO: A Millimeter-Wave Dataset and Simulator for Non-Line-of-Sight Perception".

Jane: Here's the gist of the paper. The ability to observe the world is fundamental to reasoning and making informed decisions on how to interact with the environment MITO, a novel dataset of millimeter-wave images for everyday objects, creates a foundation for pervasive non-line-of-sight perception.

Tom: Dataset Creation and Scope MITO is the first millimeter-wave (mmWave) dataset of diverse, everyday objects collected using a UR5 robotic arm with two mmWave radars operating at different frequencies and an RGB-D camera.

Jane: The dataset captures over 24 million mmWave frames and uses them to generate 550 high-resolution mmWave (synthetic aperture) images in line-of-sight and non-light-ofsight (NLOS), as well as RGB-D images, segmentation masks, and raw mmWave signals taken from 76 different objects.

Tom: The dataset builds on the YCB dataset, selecting 76 out of 80 objects to cover a diverse range of shapes, sizes, materials, and categories.

Title and authors: Tom: So we're looking at this paper now called "MITO: A Millimeter-Wave Dataset and Simulator for Non-Line-of-Sight Perception". It seems like they are tackling a big problem in perception where things get blocked, you know, occlusions.

Jane: Exactly. The core idea is that because millimeter wave signals can actually go through things like cardboard boxes or fabric, we can get information about objects even when we can't see them directly with a camera. This paper introduces MITO as the first dataset of these images for everyday objects collected using a UR5 robotic arm and two different mmWave radars.

Lu: What's interesting here is that they aren't just taking pictures; they are generating something called synthetic aperture images from over twenty-four million mmWave frames to create high-resolution images, which is key for getting those better details for non-line-of-sight perception <ref:2502.10259#pg1,for non-line-of-sight perception>.

Meng: From a production standpoint, the dataset size sounds massive, but I wonder how much of that data is actually usable without needing an enormous computational backend just to process it.

Lalam: The paper says they capture over twenty-four million mmWave frames and use them to generate five hundred fifty high-resolution mmWave images for seventy-six objects from the YCB dataset, which is a good starting point because it covers a diverse range of shapes and materials <ref:2502.10259#pg2,capture over 24 million mmWave frames and use them to generate 550>.

Tom: And they provide not just those images but also RGB-D images when the object is in line of sight, as well as ground truth segmentations, which gives researchers a lot to work with for training models.

Jane: So the main point here is establishing this dataset and simulator so that future algorithms can be adapted for non-line-of-sight perception, which is a major gap in current computer vision systems.

Lu: They also developed an open source simulator that lets you make synthetic mmWave images for any three dimensional mesh, and they modeled different material properties by building two simulation methods that you can combine.

Title and authors: Meng: That sounds ambitious. Combining specular reflections and edge reflections to simulate various materials is a complex modeling task; what's the actual cost or complexity of running those simulations at scale?

Lalam: The simulator estimates raw signals by finding visible mesh vertices, measuring the distance to those locations, and simulating the reflection based on that surface normal, which then gets summed up for each measurement location.

Tom: And they specifically address different types of reflections: specular reflections and edge reflections; they combine these using a weighted formula called Isyn alpha one alpha two Ie plus alpha two over alpha one plus alpha two Ie.

Jane: So the paper is showing how to create this synthetic data foundation, and then they immediately test its usefulness by benchmarking it for two tasks: object segmentation and shape classification in non-line-of-sight scenarios.

Lu: For the benchmark, they use SAM for object segmentation and a mmWave power based prompter during prompt point selection, which is where they combine seventy-seven gigahertz and twenty-four gigahertz images to get the best segmentation results <ref:2502.10259#pg2>.

Meng: I’m curious about that multi-spectral approach; does combining those two frequencies actually give a measurable improvement over just using one frequency when you're trying to filter out noise from things like cardboard?

Jane: The paper suggests that this multi-spectral combination allows the lower frequency image to act as a filter, which removes the impact of noise, such as what you might see from cardboard occluders during segmentation tasks.

Tom: And for shape classification, they trained a system entirely on synthetic images and showed it can still classify real world images, with their combined simulation achieving an overall accuracy of eighty-five point two percent in shape classification <ref:2502.10259#pg2>.

Lu: That eighty-five point two percent accuracy is achieved by combining both specular and edge simulations using different weights; the combination of both simulation types achieves that accuracy, which they say is superior to using just one type of simulation alone <ref:2502.10259#pg2>.

Meng: So that's a solid result on the synthetic side, but I wonder about deploying this classifier in a real environment where you don't have perfect knowledge of the material properties beforehand.

Title and authors: Jane: That’s exactly where the next part comes in, because they suggest three main improvements to push this further into practical application for non-line-of-sight perception.

Tom: The first improvement is about improving object segmentation by leveraging that multi-spectral fusion during prompt point selection with Segment Anything Model.

Lu: They propose a robust object completion network that learns missing parts of objects specifically due to specularity, aiming to generate more photo realistic non-line-of-sight images where mmWave images usually fail because they don't capture the whole object due to specularity.

Meng: Learning missing parts sounds like a lot of training data is needed for that kind of network; what is the expected computational cost or complexity if you try to run this completion network on a standard edge device?

Jane: They also look at using these mmWave images for six degrees of freedom estimation, particularly for liquid packaging, and they point toward NLOS robotic manipulation tasks like searching in clutter or analyzing pile stability.

Tom: These future directions show how this dataset and simulator can move beyond just simple object recognition into more complex physical tasks where the environment is messy.

Lu: Overall, this work provides a foundation and motivation for future research in mmWave based non-line-of-sight perception by giving us both a dataset and a simulator that establish strong baselines.

Meng: So, to wrap up, this MITO paper gives us the tools to start building systems that can reliably perceive objects through occlusions using millimeter wave radar data.

Lalam: This work provides a dataset and simulator that establish baselines for nonline-of-sight object segmentation and classification, paving the way for important applications in robotic perception, logistics, scene understanding.

Tom: That’s the big picture here—moving mmWave perception out of controlled settings into more open domains with diverse categories.

Jane: And they've made it accessible by providing both a dataset and a simulator available at github signalkinetics MITO Codebase so others can build on this foundation right away.

The paper's summary: Tom: So, this paper, MITO, boils down to them building a whole system—a dataset and a simulator—to help AI see things when there's an obstruction in the way.

Jane: That’s right. They created this collection of over twenty-four million millimeter wave frames from everyday objects using a robotic arm and two different radars, which is the foundation for this whole non-line-of-sight perception thing.

Lu: What they did then was develop an open source simulator that can make synthetic mmWave images for any three dimensional mesh. And to handle different materials, they built two ways to model it and combined them using a weighted formula.

Meng: So, the simulation part is creating fake mmWave signals based on how light bounces off the object’s surface—either purely reflecting or bouncing off the edges.

Tom: Exactly. They tested this simulator by benchmarking it for two main things: object segmentation and shape classification in those non-line-of-sight scenarios.

Jane: For segmentation, they used a model called Segment Anything Model with a multi spectral approach, combining images from both seventy seven gigahertz and twenty four gigahertz to pick the best points.

Lu: And for classification, they trained an AI system entirely on those synthetic images and showed it could still correctly classify real world pictures.

Tom: The results show that when they combine both simulation types—the specular reflections and the edge reflections—using different weights, they get eighty five point two percent accuracy in shape classification.

Jane: That eighty five point two percent is a solid number because it shows that combining those two simulation methods is actually better than using just one type of reflection simulation by itself.

Lu: The authors also suggest three big next steps for this research, including building networks to complete missing parts of objects due to specularity.

Meng: Completing missing parts sounds ambitious because mmWave images often miss things because of that specularity, so they want a network to fill in those gaps and make the synthetic images more realistic.

Tom: And they also look at using this data for estimating the six degrees of freedom from liquid packaging and tackling real world stuff like searching in clutter or analyzing pile stability.

Jane: So, this work sets up a strong baseline for non-line-of-sight perception by giving us the tools to test and train these new AI models in messy, real world environments.

Lu: The whole point is that this dataset and simulator serve as a foundation and motivation for future research in mmWave based non-line-of-sight perception.

Tom: Basically, they’ve given everyone a starting point for using millimeter wave data to understand objects even when you can't see them directly.

The paper's improvements: Tom: So, we're looking at what they think needs to happen next for this MITO work to actually be useful in the real world, and it’s pretty ambitious.

Jane: Right, they’ve laid down a strong foundation with the simulator and dataset, but now they are pushing into making the AI systems even smarter for complex tasks.

Lu: They suggest first improving object segmentation by using that multi spectral fusion during prompt point selection for the Segment Anything Model. That means combining information from both seventy seven gigahertz and twenty four gigahertz images to pick better spots.

Tom: So, they want to use the lower frequency image to act like a filter, specifically to get rid of noise from things like cardboard occluders when selecting those points for segmentation.

Jane: That makes sense. And then they propose creating an object completion network that learns the missing pieces of objects because mmWave images often don't capture the whole thing due to specularity.

Meng: Learning missing parts is tough computationally, Tom, what’s the practical cost if you have to run that completion network on a regular edge device?

Tom: It’s a huge training effort for that kind of network, but they are aiming for more photo realistic non-line-of-sight images where the object looks like it actually is.

Lu: They also look at using these mmWave images to estimate six degrees of freedom from things like liquid packaging, which is a very specific and challenging estimation task.

Jane: And they point toward more complex tasks for robotic manipulation, specifically searching in clutter or analyzing pile stability when objects are hidden.

Tom: So the paper moves beyond just simple object recognition into tackling more physically demanding perception problems where the environment is definitely not clean.

Lu: Basically, they want to see this data and simulator used to build systems that can handle those messy scenarios more reliably than before.

Jane: This shows how this work is meant to move mmWave perception out of controlled settings and into much wider domains with diverse categories of things.

Conclusion: Tom: So, to wrap up, the MITO paper gives us this dataset and simulator that sets a baseline for object segmentation and shape classification in non-line-of-sight perception using millimeter wave radar.

Jane: Exactly. It’s essentially giving researchers a powerful tool to start building systems that can perceive objects even when they are blocked from view by things like cardboard or fabric.

Lu: The implication is that we have a much more robust way to test and train AI models for these challenging perception tasks than we had before.

Meng: From my side, the simulator is good, but the real question is how much compute you need to run those simulations at scale for every possible object shape.

Lalam: I'm pretty sure this paper makes the literature on mmWave NLOS perception a lot more grounded because it provides actual data and a way to model reality.

Tom: It definitely pushes the boundary of what we thought was possible with millimeter wave data in these kinds of challenging scenarios.

Jane: This work serves as a foundation and motivation for future research in mmWave based non-line-of-sight perception, especially when we look at things like searching in clutter or figuring out stability.

Lu: We’re excited to see how this simulator feeds into those more complex tasks, like object completion networks that learn missing parts of objects due to specularity.

Meng: If those completion networks can actually be deployed efficiently without needing massive computational resources just for inference, that changes things for production systems.

Lalam: I’m fairly confident this paper is the right direction because it tackles a real physical limitation in perception, and the results are concrete.

More episodes

← Home