MITO: A Millimeter-Wave Dataset and Simulator for Non-Line-of-Sight Perception
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: We're looking at the paper: "MITO: A Millimeter-Wave Dataset and Simulator for Non-Line-of-Sight Perception".
Jane: Here's the gist of the paper. The ability to observe the world is fundamental to reasoning and making informed decisions on how to interact with the environment MITO, a novel dataset of millimeter-wave images for everyday objects, creates a foundation for pervasive non-line-of-sight perception.
Tom: Dataset Creation and Scope MITO is the first millimeter-wave (mmWave) dataset of diverse, everyday objects collected using a UR5 robotic arm with two mmWave radars operating at different frequencies and an RGB-D camera.
Jane: The dataset captures over 24 million mmWave frames and uses them to generate 550 high-resolution mmWave (synthetic aperture) images in line-of-sight and non-light-ofsight (NLOS), as well as RGB-D images, segmentation masks, and raw mmWave signals taken from 76 different objects.
Tom: The dataset builds on the YCB dataset, selecting 76 out of 80 objects to cover a diverse range of shapes, sizes, materials, and categories.
Title and authors: Tom: So we're looking at this paper now called "MITO: A Millimeter-Wave Dataset and Simulator for Non-Line-of-Sight Perception". It seems like they are tackling a big problem in perception where things get blocked, you know, occlusions.
Jane: Exactly. The core idea is that because millimeter wave signals can actually go through things like cardboard boxes or fabric, we can get information about objects even when we can't see them directly with a camera. This paper introduces MITO as the first dataset of these images for everyday objects collected using a UR5 robotic arm and two different mmWave radars.
Lu: What's interesting here is that they aren't just taking pictures; they are generating something called synthetic aperture images from over twenty-four million mmWave frames to create high-resolution images, which is key for getting those better details for non-line-of-sight perception <ref:2502.10259#pg1,for non-line-of-sight perception>.
Meng: From a production standpoint, the dataset size sounds massive, but I wonder how much of that data is actually usable without needing an enormous computational backend just to process it.
Lalam: The paper says they capture over twenty-four million mmWave frames and use them to generate five hundred fifty high-resolution mmWave images for seventy-six objects from the YCB dataset, which is a good starting point because it covers a diverse range of shapes and materials <ref:2502.10259#pg2,capture over 24 million mmWave frames and use them to generate 550>.
Tom: And they provide not just those images but also RGB-D images when the object is in line of sight, as well as ground truth segmentations, which gives researchers a lot to work with for training models.
Jane: So the main point here is establishing this dataset and simulator so that future algorithms can be adapted for non-line-of-sight perception, which is a major gap in current computer vision systems.
Lu: They also developed an open source simulator that lets you make synthetic mmWave images for any three dimensional mesh, and they modeled different material properties by building two simulation methods that you can combine.
Title and authors: Meng: That sounds ambitious. Combining specular reflections and edge reflections to simulate various materials is a complex modeling task; what's the actual cost or complexity of running those simulations at scale?
Lalam: The simulator estimates raw signals by finding visible mesh vertices, measuring the distance to those locations, and simulating the reflection based on that surface normal, which then gets summed up for each measurement location.
Tom: And they specifically address different types of reflections: specular reflections and edge reflections; they combine these using a weighted formula called Isyn alpha one alpha two Ie plus alpha two over alpha one plus alpha two Ie.
Jane: So the paper is showing how to create this synthetic data foundation, and then they immediately test its usefulness by benchmarking it for two tasks: object segmentation and shape classification in non-line-of-sight scenarios.
Lu: For the benchmark, they use SAM for object segmentation and a mmWave power based prompter during prompt point selection, which is where they combine seventy-seven gigahertz and twenty-four gigahertz images to get the best segmentation results <ref:2502.10259#pg2>.
Meng: I’m curious about that multi-spectral approach; does combining those two frequencies actually give a measurable improvement over just using one frequency when you're trying to filter out noise from things like cardboard?
Jane: The paper suggests that this multi-spectral combination allows the lower frequency image to act as a filter, which removes the impact of noise, such as what you might see from cardboard occluders during segmentation tasks.
Tom: And for shape classification, they trained a system entirely on synthetic images and showed it can still classify real world images, with their combined simulation achieving an overall accuracy of eighty-five point two percent in shape classification <ref:2502.10259#pg2>.
Lu: That eighty-five point two percent accuracy is achieved by combining both specular and edge simulations using different weights; the combination of both simulation types achieves that accuracy, which they say is superior to using just one type of simulation alone <ref:2502.10259#pg2>.
Meng: So that's a solid result on the synthetic side, but I wonder about deploying this classifier in a real environment where you don't have perfect knowledge of the material properties beforehand.
Title and authors: Jane: That’s exactly where the next part comes in, because they suggest three main improvements to push this further into practical application for non-line-of-sight perception.
Tom: The first improvement is about improving object segmentation by leveraging that multi-spectral fusion during prompt point selection with Segment Anything Model.
Lu: They propose a robust object completion network that learns missing parts of objects specifically due to specularity, aiming to generate more photo realistic non-line-of-sight images where mmWave images usually fail because they don't capture the whole object due to specularity.
Meng: Learning missing parts sounds like a lot of training data is needed for that kind of network; what is the expected computational cost or complexity if you try to run this completion network on a standard edge device?
Jane: They also look at using these mmWave images for six degrees of freedom estimation, particularly for liquid packaging, and they point toward NLOS robotic manipulation tasks like searching in clutter or analyzing pile stability.
Tom: These future directions show how this dataset and simulator can move beyond just simple object recognition into more complex physical tasks where the environment is messy.
Lu: Overall, this work provides a foundation and motivation for future research in mmWave based non-line-of-sight perception by giving us both a dataset and a simulator that establish strong baselines.
Meng: So, to wrap up, this MITO paper gives us the tools to start building systems that can reliably perceive objects through occlusions using millimeter wave radar data.
Lalam: This work provides a dataset and simulator that establish baselines for nonline-of-sight object segmentation and classification, paving the way for important applications in robotic perception, logistics, scene understanding.
Tom: That’s the big picture here—moving mmWave perception out of controlled settings into more open domains with diverse categories.
Jane: And they've made it accessible by providing both a dataset and a simulator available at github signalkinetics MITO Codebase so others can build on this foundation right away.
The paper's summary: Tom: So, this paper, MITO, boils down to them building a whole system—a dataset and a simulator—to help AI see things when there's an obstruction in the way.
Jane: That’s right. They created this collection of over twenty-four million millimeter wave frames from everyday objects using a robotic arm and two different radars, which is the foundation for this whole non-line-of-sight perception thing.
Lu: What they did then was develop an open source simulator that can make synthetic mmWave images for any three dimensional mesh. And to handle different materials, they built two ways to model it and combined them using a weighted formula.
Meng: So, the simulation part is creating fake mmWave signals based on how light bounces off the object’s surface—either purely reflecting or bouncing off the edges.
Tom: Exactly. They tested this simulator by benchmarking it for two main things: object segmentation and shape classification in those non-line-of-sight scenarios.
Jane: For segmentation, they used a model called Segment Anything Model with a multi spectral approach, combining images from both seventy seven gigahertz and twenty four gigahertz to pick the best points.
Lu: And for classification, they trained an AI system entirely on those synthetic images and showed it could still correctly classify real world pictures.
Tom: The results show that when they combine both simulation types—the specular reflections and the edge reflections—using different weights, they get eighty five point two percent accuracy in shape classification.
Jane: That eighty five point two percent is a solid number because it shows that combining those two simulation methods is actually better than using just one type of reflection simulation by itself.
Lu: The authors also suggest three big next steps for this research, including building networks to complete missing parts of objects due to specularity.
Meng: Completing missing parts sounds ambitious because mmWave images often miss things because of that specularity, so they want a network to fill in those gaps and make the synthetic images more realistic.
Tom: And they also look at using this data for estimating the six degrees of freedom from liquid packaging and tackling real world stuff like searching in clutter or analyzing pile stability.
Jane: So, this work sets up a strong baseline for non-line-of-sight perception by giving us the tools to test and train these new AI models in messy, real world environments.
Lu: The whole point is that this dataset and simulator serve as a foundation and motivation for future research in mmWave based non-line-of-sight perception.
Tom: Basically, they’ve given everyone a starting point for using millimeter wave data to understand objects even when you can't see them directly.
The paper's improvements: Tom: So, we're looking at what they think needs to happen next for this MITO work to actually be useful in the real world, and it’s pretty ambitious.
Jane: Right, they’ve laid down a strong foundation with the simulator and dataset, but now they are pushing into making the AI systems even smarter for complex tasks.
Lu: They suggest first improving object segmentation by using that multi spectral fusion during prompt point selection for the Segment Anything Model. That means combining information from both seventy seven gigahertz and twenty four gigahertz images to pick better spots.
Tom: So, they want to use the lower frequency image to act like a filter, specifically to get rid of noise from things like cardboard occluders when selecting those points for segmentation.
Jane: That makes sense. And then they propose creating an object completion network that learns the missing pieces of objects because mmWave images often don't capture the whole thing due to specularity.
Meng: Learning missing parts is tough computationally, Tom, what’s the practical cost if you have to run that completion network on a regular edge device?
Tom: It’s a huge training effort for that kind of network, but they are aiming for more photo realistic non-line-of-sight images where the object looks like it actually is.
Lu: They also look at using these mmWave images to estimate six degrees of freedom from things like liquid packaging, which is a very specific and challenging estimation task.
Jane: And they point toward more complex tasks for robotic manipulation, specifically searching in clutter or analyzing pile stability when objects are hidden.
Tom: So the paper moves beyond just simple object recognition into tackling more physically demanding perception problems where the environment is definitely not clean.
Lu: Basically, they want to see this data and simulator used to build systems that can handle those messy scenarios more reliably than before.
Jane: This shows how this work is meant to move mmWave perception out of controlled settings and into much wider domains with diverse categories of things.
Conclusion: Tom: So, to wrap up, the MITO paper gives us this dataset and simulator that sets a baseline for object segmentation and shape classification in non-line-of-sight perception using millimeter wave radar.
Jane: Exactly. It’s essentially giving researchers a powerful tool to start building systems that can perceive objects even when they are blocked from view by things like cardboard or fabric.
Lu: The implication is that we have a much more robust way to test and train AI models for these challenging perception tasks than we had before.
Meng: From my side, the simulator is good, but the real question is how much compute you need to run those simulations at scale for every possible object shape.
Lalam: I'm pretty sure this paper makes the literature on mmWave NLOS perception a lot more grounded because it provides actual data and a way to model reality.
Tom: It definitely pushes the boundary of what we thought was possible with millimeter wave data in these kinds of challenging scenarios.
Jane: This work serves as a foundation and motivation for future research in mmWave based non-line-of-sight perception, especially when we look at things like searching in clutter or figuring out stability.
Lu: We’re excited to see how this simulator feeds into those more complex tasks, like object completion networks that learn missing parts of objects due to specularity.
Meng: If those completion networks can actually be deployed efficiently without needing massive computational resources just for inference, that changes things for production systems.
Lalam: I’m fairly confident this paper is the right direction because it tackles a real physical limitation in perception, and the results are concrete.
Massachusetts Institute of Technology · Adobe Research
cs.CV
Submitted: 2025-02-14
Updated: 2026-11-05
Code: https://github.com/signalkinetics/MITO
License: http://creativecommons.org/licenses/by-nc-sa/4.0/
Importance score: 96/100
The gist: The ability to observe the world is fundamental to reasoning and making informed decisions on how to interact with the environment MITO…
Key concepts
- MITO Dataset
- This is a large collection of millimeter-wave (mmWave) images of 76 diverse everyday objects. It was captured using a UR5 robot with two different mmWave radars and an RGB-D camera, providing high-resolution data for studying non-line-of-sight perception.
- Simulation Framework
- An open-source simulator is developed to create synthetic mmWave images from any 3D shape. It models material properties by simulating reflections based on distance and surface normals, including variations for specular and edge reflections.
- Non-Line-of-Sight (NLOS) Perception
- This refers to the ability of a system to perceive objects even when they are not directly visible. The MITO framework is designed to improve this capability by using multi-spectral fusion and advanced simulation techniques for object segmentation and classification.
- Multi-spectral Fusion
- This technique combines information from two different mmWave frequencies (77 GHz and 24 GHz) during prompt point selection for object segmentation. This helps the system better filter out noise, improving accuracy in identifying objects when they are partially occluded.
Terminology
Summary
The ability to observe the world is fundamental to reasoning and making informed decisions on how to interact with the environment MITO, a novel dataset of millimeter-wave images for everyday objects, creates a foundation for pervasive non-line-of-sight perception. p. 0 edu
1 Massachusetts Institute of Technology, 2 Adobe Research, 3 Cartesian Systems
* These authors contributed equally
Abstract
The ability to observe the world is fundamental to reasoning and making informed decisions on how to interact with
the environment. However, optical perception can often be
disrupted due to common occurrences, such as occlusions,
which can pose challenges to existing vision systems.Where the paper says it
Dataset Creation and Scope
MITO is the first millimeter-wave (mmWave) dataset of diverse, everyday objects collected using a UR5 robotic arm with two mmWave radars operating at different frequencies and an RGB-D camera. The dataset captures over 24 million mmWave frames and uses them to generate 550 high-resolution mmWave (synthetic aperture) images in line-of-sight and non-light-ofsight (NLOS), as well as RGB-D images, segmentation masks, and raw mmWave signals taken from 76 different objects. p. 0 To capture higher-resolution mmWave images, we leverage the
robot’s mobility and fuse frames over the synthesized aperture. MITO captures over 24 million mmWave frames and
uses them to generate 550 high-resolution mmWave (synthetic aperture) images in line-of-sight and non-light-ofsight (NLOS), as well as RGB-D images, segmentation
masks, and raw mmWave signals, taken from 76 different
objects. We select
76 out of 80 objects from the YCB dataset (the remaining
4 were discarded due to not being able to obtain the exact
object). Unlike previous mmWave datasets, these objects
cover a diverse range of shapes, sizes, materials, and categories.Where the paper says it
Where the paper says it
Simulation Framework Development
The work includes the development of an open-source simulator that can produce synthetic mmWave images for any 3D triangle mesh. To model different material properties, two different modeling methods are built into the simulation, which can be combined to simulate a wide range of material properties. p. 1 We develop an open-source simulator that
can produce synthetic mmWave images for any 3D triangle mesh. Since different materials interact differently
with mmWave signals and, therefore, produce different
images, we build two different modeling methods into our
simulation, the combination of which can be used to simulate a wide range of material properties. Given the distance, we
simulate the reflection from this vertex:
sj (l, v) = e
−j
4π|l−v|
λj (4)
where sj (l, v) is the jth sample of the reflection from location l to vertex v and back. Finally, we sum the simulated reflections from all visible
vertices to produce the raw signal for this measurement
location. To accurately model different objects, we introduce 2 variations of
the simulation to account for different types of reflections. Specular Reflections
When an object exhibits purely specular reflections, the signal will reflect off each vertex in only one direction.Where the paper says it
Where the paper says it
Where the paper says it
Perception Benchmarking
The utility of MITO is demonstrated through benchmarks for two computer vision tasks in nonline-of-sight (NLOS) perception.
-
Object segmentation is shown using the segment-anything model (SAM) and a mmWave power-based prompter. p. 1
Where the paper says it
Using MITO, we establish baselines for two computer vision tasks in nonline-of-sight. First, we show object segmentation, using the segment-anything model (SAM) [27] and a mmWave power-based prompter.
-
Shape classification is trained entirely on synthetic images and can classify real-world images. p. 1
Where the paper says it
First, we show object segmentation, using the segment-anything model (SAM) [27] and a mmWave power-based prompter. Second, we show shape classification, which is trained entirely on synthetic images and can classify real-world images.
The multi-spectral approach, combining 77 GHz and 24 GHz images during prompt point selection, achieves the highest performance in segmentation. p. 6 In particular, the
F-score of NLOS for 24 GHz, 77 GHz, and multi-spectral
exceeds that of the camera. Notably, multi-spectral images
achieve the highest performance.Where the paper says it
Performance Evaluation
The simulator's accuracy is evaluated using the standard 3D F-score metric to compare mmWave outputs from real-world and simulation. The median F-score for MITO's simulator is 94%, while specular-only or edge-only simulators achieve median F-scores of 90% and 71%, respectively. The combined simulation, using both specular and edge simulations with different weights, achieves an overall accuracy of 85. p. 7 2%
Table 3. Classifier Accuracy for LOS, NLOS,& overall
using specular, edge,& combined simulations for training. First, we show object segmentation, using
the segment-anything model (SAM) [27] and a mmWave
power-based prompter. Second, we show shape classification, which is trained entirely on synthetic images and
can classify real-world images.Where the paper says it
Where the paper says it
Future Research Directions
The paper suggests several avenues for future work to expand the scope of NLOS perception. These include NLOS Object Completion networks to learn missing parts, 6DoF estimation from mmWave images for liquid packaging, and NLOS Robotic Manipulation tasks such as searching in clutter or analyzing pile stability. p. 7 It would be interesting future work to create an object completion network
that learns the missing parts of an object & creates more
photo-realistic NLOS images. • NLOS 6DoF Estimation: Another interesting problem is
6DoF estimation from mmWave images. foundation and motivation for future research in mmWavebased NLOS perception, similar to how early RGB datasets
accelerated the research in optical-based Computer Vision,
and continued driving the field forward as they expanded. References
[1] Ti dca1000evm.Where the paper says it
Where the paper says it
Conclusion
MITO provides a dataset and simulator that establish baselines for NLOS object segmentation and classification, paving the way for important applications in robotic perception, logistics, scene understanding. The research aims to expand the application of mmWave perception to a more open domain consisting of diverse categories. p. 1 However, these approaches are
limited to specific domains, such as vehicles [22, 29, 42],
pedestrians [24], weapons [39, 40], or entirely metallic objects [30]. Through a new dataset and simulator, our work
aims to expand the application of mmWave perception to a
more open domain consisting of diverse categories. Second, we show shape classification, which is trained entirely on synthetic images and
can classify real-world images. The dataset, code, and a video demonstration are available here:
https://github. The dataset, code, and a video demonstration are available here:
https://github. com/signalkinetics/MITO Codebase
2. foundation and motivation for future research in mmWavebased NLOS perception, similar to how early RGB datasets
accelerated the research in optical-based Computer Vision,
and continued driving the field forward as they expanded. References
[1] Ti dca1000evm. foundation and motivation for future research in mmWavebased NLOS perception, similar to how early RGB datasets
accelerated the research in optical-based Computer Vision,
and continued driving the field forward as they expanded. References
[1] Ti dca1000evm.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
References
[1] Ti dca1000evm. https: / / www. p. 8 References
[1] Ti dca1000evm. https : / / www . https : / / www . com / tool /
DCA1000EVM.Where the paper says it
Where the paper says it
[3] Perceiving systems mesh package. https://ps. p. 1 Second, we show shape classification, which is trained entirely on synthetic images and
can classify real-world images. The dataset, code, and a video demonstration are available here:
https://github.Where the paper says it
[4] Infineon position2go. https://www. p. 8 References
[1] Ti dca1000evm. https : / / www . References
[1] Ti dca1000evm. https : / / www . References
[1] Ti dca1000evm. https : / / www . References
[1] Ti dca1000evm. https : / / www . https://www. com/
cms/en/product/evaluation- boards/demoposition2go/.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
[6] Ti iwr1443boost. https://www.ti.com /product / IWR1443tech-docs. p. 8 https://www. com/product/
IWR1443#tech-docs.Where the paper says it
[7] mmWave Radar Radome Design Guide. https://www.ti.com/lit/an/swra705/swra705. p. 8 https://www. com/lit/an/swra705/swra705.Where the paper says it
[8] Ansys high frequency structure simulator (hfss), 2022. Ansys HFSS. p. 8 Texas Instruments. 4
[8] Ansys high frequency structure simulator (hfss), 2022.Where the paper says it
[9] Fadel Adib, Zach Kabelac, Dina Katabi, and Robert C Miller. 3d tracking via body radio reflections. p. 8 2
[9] Fadel Adib, Zach Kabelac, Dina Katabi, and Robert C Miller. 3d tracking via body radio reflections.Where the paper says it
[10] Fadel Adib, Chen-Yu Hsu, Hongzi Mao, Dina Katabi, and Fredo Durand. Capturing the human figure through a wall. p. 8 3
[10] Fadel Adib, Chen-Yu Hsu, Hongzi Mao, Dina Katabi, and
Fredo Durand. Capturing the human figure through a wall.Where the paper says it
[11] Mario Bijelic, Tobias Gruber, Fahim Mannan, Florian Kraus, Werner Ritter, Klaus Dietmayer, and Felix Heide. Seeing through fog without seeing fog: Deep multimodal sensor fusion in unseen adverse weather. p. 8 4
[11] Mario Bijelic, Tobias Gruber, Fahim Mannan, Florian Kraus,
Werner Ritter, Klaus Dietmayer, and Felix Heide. Seeing
through fog without seeing fog: Deep multimodal sensor fusion in unseen adverse weather.Where the paper says it
[14] Berk Calli, Arjun Singh, James Bruce, Aaron Walsman, Kurt Konolige, Siddhartha Srinivasa, Pieter Abbeel, and Aaron M Dollar. Yale-cmu-berkeley dataset for robotic manipulation research. p. 8 2
[14] Berk Calli, Arjun Singh, James Bruce, Aaron Walsman, Kurt
Konolige, Siddhartha Srinivasa, Pieter Abbeel, and Aaron M
Dollar. Yale-cmu-berkeley dataset for robotic manipulation
research.Where the paper says it
[15] Anjun Chen, Xiangyu Wang, Shaohao Zhu, Yanxu Li, Jiming Chen, and Qi Ye. mmbody benchmark: 3d body reconstruction dataset and analysis for millimeter wave radar, 2022
[16] Han Cui, Shu Zhong, Jiacheng Wu, Zichao Shen, Naim Dahnoun, and Yiren Zhao. Milipoint: A point cloud dataset for mmwave radar, 2023
[17] Pham Tien Dat, Yuya Yamaguchi, Shingo Takano, Shotaro Hirata, Junichiro Ichikawa, Ryo Shimizu, Keizo Inagaki, Isao Morohashi, Yuki Yoshida, Atsushi Kanno, Naokatsu Yamamoto, Tetsuya Kawanishi, and Kouichi Akahane. Terahertz signal transparent relay and switching using photonic technology. p. 8 2
[17] Pham Tien Dat, Yuya Yamaguchi, Shingo Takano, Shotaro
Hirata, Junichiro Ichikawa, Ryo Shimizu, Keizo Inagaki,
Isao Morohashi, Yuki Yoshida, Atsushi Kanno, Naokatsu Yamamoto, Tetsuya Kawanishi, and Kouichi Akahane. Terahertz signal transparent relay and switching using photonic
technology.Where the paper says it
[18] Laura Dodds, Hailan Shanbhag, Junfeng Guan, Saurabh Gupta, and Haitham Hassanieh. Around the corner mmwave imaging in practical environments. p. 8 3
[18] Laura Dodds, Hailan Shanbhag, Junfeng Guan, Saurabh
Gupta, and Haitham Hassanieh. Around the corner mmwave
imaging in practical environments.Where the paper says it
[19] M. Everingham, L. p. 8 In Proceedings of the
30th Annual International Conference on Mobile Computing
and Networking, pages 1–15, 2024. Everingham, L. Everingham, L. Van Gool, C. Van Gool, C. Williams, J. Williams, J. Winn,
and A. Winn,
and A. The pascal visual object classes (voc)
challenge.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
[20] Federal Communications Commission. Frequency Allocation:. p. 8 2
[20] Federal Communications Commission. Frequency Allocation:. Frequency Allocation:. transition. transition. gov/oet/spectrum/
table/fcctable.Where the paper says it
Where the paper says it
Where the paper says it
[21] Simon Fothergill, Helena Mentis, Pushmeet Kohli, and Sebastian Nowozin. Instructing people for training gestural interactive systems. p. 8 3
[21] Simon Fothergill, Helena Mentis, Pushmeet Kohli, and Sebastian Nowozin. Instructing people for training gestural interactive systems.Where the paper says it
[22] Junfeng Guan, Sohrab Madani, Suraj Jog, Saurabh Gupta, and Haitham Hassanieh. Through fog high-resolution imaging using millimeter wave radar. p. 8 2
[22] Junfeng Guan, Sohrab Madani, Suraj Jog, Saurabh Gupta,
and Haitham Hassanieh. Through fog high-resolution imaging using millimeter wave radar.Where the paper says it
[23] Runwei Guan, Ruixiao Zhang, Ningwei Ouyang, Jianan Liu, Ka Lok Man, Xiaohao Cai, Ming Xu, Jeremy Smith, Eng Gee Lim, Yutao Yue, et al. Talk2radar: Bridging natural language with 4d mmwave radar for 3d referring expression comprehension. p. 8 2
[23] Runwei Guan, Ruixiao Zhang, Ningwei Ouyang, Jianan
Liu, Ka Lok Man, Xiaohao Cai, Ming Xu, Jeremy Smith,
Eng Gee Lim, Yutao Yue, et al. Talk2radar: Bridging natural
language with 4d mmwave radar for 3d referring expression
comprehension. Talk2radar: Bridging natural
language with 4d mmwave radar for 3d referring expression
comprehension. arXiv preprint arXiv:2405.Where the paper says it
Where the paper says it
[27] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollar, and Ross Girshick. p. 8 2
[27] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao,
Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollar, and ´ Using MITO, we establish baselines for two computer vision tasks in nonline-of-sight. First, we show object segmentation, using
the segment-anything model (SAM) [27] and a mmWave
power-based prompter. Segment anything. arXiv:2304.Where the paper says it
Where the paper says it
Where the paper says it
[28] Andrew Kramer, Kyle Harlow, Christopher Williams, and Christoffer Heckman. Coloradar: The direct 3d millimeter wave radar dataset. p. 9 2, 4, 6, 7
[28] Andrew Kramer, Kyle Harlow, Christopher Williams, and
Christoffer Heckman. Coloradar: The direct 3d millimeter
wave radar dataset.Where the paper says it
[29] Mohamed Lamane, Mohamed Tabaa, and Abdessamad Klilou. Classification of targets detected by mmwave radar using yolov5. p. 9 2
[29] Mohamed Lamane, Mohamed Tabaa, and Abdessamad
Klilou. Classification of targets detected by mmwave radar
using yolov5.Where the paper says it
[30] Jie Liu, Kai Zhang, Zhenlin Sun, Qiang Wu, Wei He, and Hao Wang. Concealed object detection and recognition system based on millimeter wave fmcw radar. p. 9 2
[30] Jie Liu, Kai Zhang, Zhenlin Sun, Qiang Wu, Wei He, and
Hao Wang. Concealed object detection and recognition system based on millimeter wave fmcw radar.Where the paper says it
[31] Ting Liu, Yao Zhao, Yunchao Wei, Yufeng Zhao, and Shikui Wei. Concealed object detection for activate millimeter wave image. p. 9 2
[31] Ting Liu, Yao Zhao, Yunchao Wei, Yufeng Zhao, and Shikui
Wei. Concealed object detection for activate millimeter wave
image.Where the paper says it
[32] Jonathan S. Lu, Patrick Cabrol, Daniel Steinbach, and Ravikumar V. p. 9 2
[32] Jonathan S. Lu, Patrick Cabrol, Daniel Steinbach, and
Ravikumar V. Lu, Patrick Cabrol, Daniel Steinbach, and
Ravikumar V. Measurement and characterization
of various outdoor 60 ghz diffracted and scattered paths.Where the paper says it
Where the paper says it
[33] Marco Mezzavilla, Menglei Zhang, Michele Polese, Russell Ford, Sourjya Dutta, Sundeep Rangan, and Michele Zorzi. End-to-end simulation of 5g mmwave networks. p. 9 3
[33] Marco Mezzavilla, Menglei Zhang, Michele Polese, Russell
Ford, Sourjya Dutta, Sundeep Rangan, and Michele Zorzi. End-to-end simulation of 5g mmwave networks.Where the paper says it
[34] Dong-Hee Paek, Seung-Hyun Kong, and Kevin Tirta Wijaya. K-radar: 4d radar object detection for autonomous driving in various weather conditions. p. 9 2
[34] Dong-Hee Paek, Seung-Hyun Kong, and Kevin Tirta Wijaya. K-radar: 4d radar object detection for autonomous driving
in various weather conditions.Where the paper says it
[35] Andras Palffy, Ewoud Pool, Srimannarayana Baratam, Julian F. P. Kooij, and Dariu M. p. 9 2
[35] Andras Palffy, Ewoud Pool, Srimannarayana Baratam, Julian F. Kooij, and Dariu M. Kooij, and Dariu M. Multi-class road user
detection with 3+1d radar in the view-of-delft dataset.Where the paper says it
Where the paper says it
[36] Akarsh Prabhakara, Tao Jin, Arnav Das, Gantavya Bhatt, Lilly Kumari, Elahe Soltanaghai, Jeff Bilmes, Swarun Kumar, and Anthony Rowe. Radarhd: Demonstrating lidar-like point clouds from mmwave radar. p. 9 2
[36] Akarsh Prabhakara, Tao Jin, Arnav Das, Gantavya Bhatt,
Lilly Kumari, Elahe Soltanaghai, Jeff Bilmes, Swarun Kumar, and Anthony Rowe. Radarhd: Demonstrating lidar-like
point clouds from mmwave radar.Where the paper says it
[37] M Mahbubur Rahman, Ryoma Yataka, Sorachi Kato, Pu Perry Wang, Peizhao Li, Adriano Cardace, and Petros Boufounos. Mmvr: Millimeter-wave multi-view radar dataset and benchmark for indoor perception. p. 9 2
[37] M Mahbubur Rahman, Ryoma Yataka, Sorachi Kato,
Pu Perry Wang, Peizhao Li, Adriano Cardace, and Petros Boufounos. Mmvr: Millimeter-wave multi-view radar
dataset and benchmark for indoor perception. Mmvr: Millimeter-wave multi-view radar
dataset and benchmark for indoor perception. arXiv preprint
arXiv:2406.Where the paper says it
Where the paper says it
[38] Hem Regmi, Moh Sabbir Saadat, Sanjib Sur, and Srihari Nelakuditi. Squigglemilli: Approximating sar imaging on mobile millimeter-wave devices. p. 9 2
[38] Hem Regmi, Moh Sabbir Saadat, Sanjib Sur, and Srihari
Nelakuditi. Squigglemilli: Approximating sar imaging on
mobile millimeter-wave devices. Squigglemilli: Approximating sar imaging on
mobile millimeter-wave devices. ACM Interact. ACM Interact. Wearable Ubiquitous Technol.Where the paper says it
Where the paper says it
Where the paper says it
[39] Bofeng Su and Minghui Yuan. Object recognition for millimeter wave mimo-sar images based on high-resolution feature recursive alignment fusion network. p. 9 2
[39] Bofeng Su and Minghui Yuan. Object recognition for millimeter wave mimo-sar images based on high-resolution feature recursive alignment fusion network.Where the paper says it
[40] Peng Sun, Ting Liu, Xiaotong Chen, Shiyin Zhang, Yao Zhao, and Shikui Wei. Multi-source aggregation transformer for concealed object detection in millimeter-wave images. p. 9 2
[40] Peng Sun, Ting Liu, Xiaotong Chen, Shiyin Zhang, Yao
Zhao, and Shikui Wei. Multi-source aggregation transformer
for concealed object detection in millimeter-wave images.Where the paper says it
[41] Shu Sun, George R MacCartney, and Theodore S Rappaport. A novel millimeter-wave channel simulator and applications for 5g wireless communications. p. 9 2
[41] Shu Sun, George R MacCartney, and Theodore S Rappaport. A novel millimeter-wave channel simulator and applications for 5g wireless communications.Where the paper says it
[43] Maxim Tatarchenko, Stephan R. Richter, Rene Ranftl, Zhuwen Li, Vladlen Koltun, and Thomas Brox. p. 9 2
[43] Maxim Tatarchenko*, Stephan R. Richter*, Rene Ranftl, ´
Zhuwen Li, Vladlen Koltun, and Thomas Brox. Richter*, Rene Ranftl, ´
Zhuwen Li, Vladlen Koltun, and Thomas Brox. What do
single-view 3d reconstruction networks learn?Where the paper says it
Where the paper says it
[44] Universal Robots, UR5e. https://www.universalrobots. p. 9 https://www. universalrobots.Where the paper says it
[46] Zhongce Wang, Chao Jin, Lingzhang Sun, and Yunfan Li. Wrdi: A multimodal dataset of mmwave radar data and image. p. 9 2
[46] Zhongce Wang, Chao Jin, Lingzhang Sun, and Yunfan Li. Wrdi: A multimodal dataset of mmwave radar data and image.Where the paper says it
[47] Sahil Waqar and Matthias Patzold. A simulation-based framework for the design of human activity recognition systems using radar sensors. p. 9 2
[47] Sahil Waqar and Matthias Patzold. A simulation-based ¨
framework for the design of human activity recognition systems using radar sensors.Where the paper says it
[48] Yingxiao Wu, Zhongmin Jiang, Haocheng Ni, Changlin Mao, Zhiyuan Zhou, Wenxiang Wang, and Jianping Han. mmhpe: Robust multi-scale 3d human pose estimation using a single mmwave radar. p. 9 2
[48] Yingxiao Wu, Zhongmin Jiang, Haocheng Ni, Changlin
Mao, Zhiyuan Zhou, Wenxiang Wang, and Jianping Han. mmhpe: Robust multi-scale 3d human pose estimation using
a single mmwave radar.Where the paper says it
[49] Jianjian Yang, Tianmu Gui, Yibo Tong, Yuyuan Zhang, Qiankun Huang, and Guanghui Zhao. Autonomous mining transportation systems: Integrating 4d mmwave radar for enhanced detection of obstructed static objects. p. 9 2
[49] Jianjian Yang, Tianmu Gui, Yibo Tong, Yuyuan Zhang,
Qiankun Huang, and Guanghui Zhao. Autonomous mining
transportation systems: Integrating 4d mmwave radar for enhanced detection of obstructed static objects.Where the paper says it
[50] Xiaotong Zhang, Zhenjiang Li, and Jin Zhang. Synthesized millimeter-waves for human motion sensing. p. 9 2
[50] Xiaotong Zhang, Zhenjiang Li, and Jin Zhang. Synthesized
millimeter-waves for human motion sensing.Where the paper says it
[51] Lianqing Zheng, Zhixiong Ma, Xichan Zhu, Bin Tan, Sen Li, Kai Long, Weiqi Sun, Sihan Chen, Lu Zhang. Tj4dradset: A 4d radar dataset for autonomous driving. p. 9 2
[51] Lianqing Zheng, Zhixiong Ma, Xichan Zhu, Bin Tan, Sen Li,
Kai Long, Weiqi Sun, Sihan Chen, Lu Zhang, Mengyue Wan,
Libo Huang, and Jie Bai. Tj4dradset: A 4d radar dataset for
autonomous driving.Where the paper says it
[51] Junyu Zhou and Julien Le Kernec. 4d radar simulator for human activity recognition. p. 9 2
[47] Sahil Waqar and Matthias Patzold. A simulation-based ¨
framework for the design of human activity recognition systems using radar sensors.Where the paper says it
How the training signal is built
The process of generating synthetic images involves estimating raw signals expected at each location given reflections off the target object. This estimation follows a three-step procedure:
-
First, find which mesh vertices in the mesh are visible from that location. p. 4
Where the paper says it
Estimating raw signals at a given location is as follows: 1. First, we find which mesh vertices in the mesh are visible from that location [3].
-
Next, for a given visible vertex, measure the distance to the measurement location and simulate reflection from this vertex. p. 4
Where the paper says it
Next, for a given visible vertex, we measure the distance to the measurement location. Given the distance, we simulate the reflection from this vertex: sj (l, v) = e −j 4π|l−v| λj (4) where sj (l, v) is the jth sample of the reflection from location l to vertex v and back.
-
Finally, sum the simulated reflections from all visible vertices to produce the raw signal for this measurement location. p. 4
Where the paper says it
Given the distance, we simulate the reflection from this vertex: sj (l, v) = e −j 4π|l−v| λj (4) where sj (l, v) is the jth sample of the reflection from location l to vertex v and back. Finally, we sum the simulated reflections from all visible vertices to produce the raw signal for this measurement location.
To model different materials, two simulation variations are used:
** Specular Reflections**
When an object exhibits purely specular reflections, a reflection is only added if the angle between the normal and the vector from radar to vertex is greater than a threshold τ.
** Edge Reflections**
For edge reflections, only reflections from edge vertices are allowed in the simulation.
The two simulation types are combined using the formula Isyn(α1, α2) = α1 / (α1 + α2) Ie + α2 / (α1 + α2) Ie where Isyn(α1, α2) is the combined synthetic image for weights. p. 5 Combing Reflections
In practice, materials rarely exhibit only one type of reflection, but instead some combination of the two. To account
for this, our synthetic images can be added together with
different weights to simulate different materials:
Isyn(α1, α2) = α1
α1 + α2
Is +
α2
α1 + α2
Ie (8)
where Isyn(α1, α2) is the combined synthetic image for
weights α1, α2, and Is and Ie are the images from the specular and edge simulations, respectively.Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Improve non-line-of-sight (NLOS) object segmentation by leveraging multi-spectral fusion during prompt point selection for Segment Anything Model (SAM). This method combines thresholds from both 24 GHz and 77 GHz images to improve segmentation accuracy, as it
allows the 24 GHz image to act as a filter, removing the impact of noise (e.g., from cardboard occluders) on the point selection.
-
Develop a shape classification system trained entirely on synthetic mmWave images that combines specular and edge reflection simulations using weighted combinations. This allows the classifier to achieve an overall accuracy of 85.2% by
using both simulation types together achieves 85.2% accuracy,
which is superior to using only one type of simulation. -
Create a robust object completion network for NLOS perception that learns missing parts of objects due to specularity. This system can generate
more photo-realistic NLOS images
by addressing the limitation wheremmWave images do not capture the entire object due to specularity.
Abstract
The ability to observe the world is fundamental to reasoning and making informed decisions on how to interact with the environment. However, optical perception can often be disrupted due to common occurrences like occlusions, which can pose challenges to existing vision systems. We present MITO, the first millimeter-wave (mmWave) dataset of diverse, everyday objects, collected using a UR5 robotic arm with two mmWave radars operating at different frequencies and an RGB-D camera. Unlike visible light, mmWave signals can penetrate common occlusions (e.g., cardboard boxes, fabric, plastic). MITO captures over 24 million mmWave frames and uses them to generate 550 high-resolution mmWave images in line-of-sight and non-light-of-sight (NLOS), as well as RGB-D images, segmentation masks, and raw mmWave signals, taken from 76 different objects. We develop an open-source simulation tool that can be used to generate synthetic mmWave images for any 3D triangle mesh. Finally, we establish benchmarks for NLOS segmentation and classification to demonstrate the utility of our dataset and simulator for enabling broader NLOS perception.
Sources
- Talk2Radar: Bridging Natural Language with 4D mmWave Radar for 3D Referring Expression Comprehension
- Segment Anything
- MMVR: Millimeter-wave Multi-View Radar Dataset and Benchmark for Indoor Perception
Related papers
- TEA: Text Encoder Alignment for Robust Concept Erasure in Text-to-Image Models
- Do Egocentric Video-Language Models Capture Both Hand- and Object-Centric Cues?
- Aura: Consistent Multi-Subject Video Generation via VLM-Grounded Semantic Alignment
- PVRF: All-in-one Adverse Weather Removal via Prior-modulated and Velocity-constrained Rectified Flow
- DLWM: Diverse Latent World Models for Efficient Multimodal Reasoning
- Out of Sight, Still in Mind: Token Compression for Omni-LLMs