IMU4D: 4D Human-Object Understanding from Wearable IMUs
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: I'm Tom.
Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.
Tom: We're looking at the paper: "IMU4D: 4D Human-Object Understanding from Wearable IMUs".
Jane: Here's the gist of the paper. Understanding human activities and their surrounding environments typically relies on visual perception, yet cameras pose persistent challenges in privacy, safety, energy efficiency, and scalability.
Tom: This paper introduces IMU-to-4D as a framework that reconstructs detailed human motion and 3D scene layouts purely from everyday wearable sensors.
Jane: Core Framework and Insight The key insight behind IMU-to-4D is that "motion, human activities, and environment are inherently coupled—everyday inertial traces encode not only how the body moves but also what surrounds it".
Tom: The framework repurposes large language models (LLMs) as spatiotemporal reasoners to jointly model the distribution of sequential inertial signals, human motions, 3D layouts, and semantic activities.
Paper summary: Tom: So we're talking about this paper today. It's called "IMU4D: 4D Human-Object Understanding from Wearable IMUs" <ref:2604.21926#pg1>. The authors are Hao-Yu Hsu, Tianhang Cheng, Jing Wen, Alexander G. Schwing, and Shenlong Wang from the University of Illinois at Urbana-Champaign.
Jane: And what this paper does is tackle the problem that understanding human activities and their surroundings usually needs cameras or vision. But those cameras have these persistent problems with privacy, safety issues, energy use, and scalability concerns <ref:2604.21926#pg1>. This paper explores an alternative: 4D perception without relying on vision <ref:2604.21926#pg1,an alternative: 4D perception without>.
Lu: The core idea is to reconstruct detailed human motion and the three dee scene layouts purely from everyday wearable sensors <ref:2604.21926#pg1,3D scene layouts purely from everyday wearable sensors>. They introduce a framework called IMU-to-4D that repurposes large language models for non-visual spatiotemporal understanding of human-scene dynamics <ref:2604.21926#pg2>.
Meng: So, the thesis here is that motion, human activities, and environment are inherently coupled—everyday inertial data encodes not just how the body moves but also what surrounds it <ref:2604.21926#pg3>. They claim that this allows the model to infer coherent, context-aware 4D representations from as few as two or three inertial sensors <ref:2604.21926#pg1>.
Tom: That sounds like a big claim for using just IMUs. What's the actual mechanics of how they make that happen? How do we get those 4D models from just inertial data <ref:2604.21926#pg1>?
Jane: They achieve this by using large language models as spatiotemporal reasoners to jointly model the distribution of sequential inertial signals, human motions, three dee layouts, and semantic activities <ref:2604.21926#pg3>. This unified formulation is what allows them to infer that context-aware data.
Lu: To make this work practically, they use a novel tokenization scheme tailored for each modality. For IMU signals, raw signals are divided into non-overlapping windows which are VQquantized using a one-dimensional VQ-VAE with non-uniform binning to yield compact discrete root tokens <ref:2604.21926#pg3>.
Meng: And for the other parts, they handle human poses continuously and scene layouts by tokenizing them with one-hot labels for object classes <ref:2604.21926#pg3>. They also use the tokenizer from a pretrained Phi-one point five model to handle text descriptions <ref:2604.21926#pg3>.
Tom: So, they're turning raw sensor data into these discrete tokens and feeding it into an LLM structure? That sounds like a lot of work just to get the input ready. What's the training setup for this joint reasoning model?
Jane: They use a two-stage training strategy to build up this foundation. First, Stage one is IMU-to-MotionText Pretraining <ref:2604.21926#pg2>. The model is trained jointly to predict human motions and text descriptions from IMU signals using a large-scale motion-text dataset fifteen thirty-seven with synthesized IMU data <ref:2604.21926#pg3>.
Lu: Stage two is the IMU-to-4D Finetuning <ref:2604.21926#pg2>. Here, the pretrained model is separately finetuned on each of their specific three dee datasets for domain adaptation to jointly optimize all the modality-specific heads and the transformer backbone <ref:2604.21926#pg3>.
Paper summary: Meng: From a production standpoint, I'm wondering about scalability. How does this framework handle running this model efficiently once it's deployed? What are the practical costs for inference?
Jane: They show that this joint reasoning framework outperforms pipelined state-of-the-art methods. Table one shows that IMU-to-4D, their method, beats everything else in both motion and text evaluation metrics <ref:2604.21926#pg3>. It also enables ambient scene understanding without vision <ref:2604.21926#pg1>.
Tom: So it beats the modular baselines across all those metrics, which is pretty strong evidence for the joint reasoning approach. What kind of specific results are they seeing in terms of physical consistency?
Jane: They found that IMU-to-4D yields more coherent and temporally stable results than prior approaches <ref:2604.21926#pg3>. They even successfully performed three dee re-localization on a dataset called ParaHome <ref:2604.21926#pg1>, which shows they have a comprehensive understanding of both the three dee scene and activities within it.
Lu: The contribution here is introducing the first framework that recovers 4D human motion, scene layouts, and textual descriptions from just three IMU sensors by repurposing LLMs for cross-modal structural reasoning <ref:2604.21926#pg3>. That's a significant architectural move.
Meng: I still have some reservations about the input quality. The paper mentions that certain ambiguities remain because the sensory input is limited, specifically that they may struggle to distinguish between similar objects or activities when relying solely on IMU signals <ref:2604.21926#pg3>. That's a real constraint for deploying this in the real world.
Jane: That's a fair point. The authors acknowledge those ambiguities exist because the input is limited, and they suggest that resolving those ambiguities would require additional input signals or longer time spans in future work <ref:2604.21926#pg3>. Plus, they don't explicitly enforce penetration constraints, which could lead to interpenetrations in the predicted three dee scenes <ref:2604.21926#pg1>.
Tom: So, what's the big picture implication here for how we think about human-computer interaction? What does this mean for someone who isn't a machine learning expert?
Lu: It paves the way for embodied AI that perceives the world through motion rather than sight <ref:2604.21926#pg3>. It suggests that wearable motion sensors alone can support rich 4D understanding, which is a big step away from vision-based perception <ref:2604.21926#pg1>.
Jane: Essentially, this means that everyday motion sensors embedded in earbuds, watches, or smartphones could reveal a lot about our activities and environments <ref:2604.21926#pg3>. It moves us toward systems that can understand context without needing a camera pointed at everything.
Meng: But for me, it's about the deployment hurdle. If you have to run this kind of joint reasoning model on real traffic or continuous streams, the computational demands might be high, and I need to see how stable those three dee reconstructions are under real-world noise <ref:2604.21926#pg1>.
Tom: Well, they did show temporal stability is better than prior work <ref:2604.21926#pg3>. They also designed a method that incorporates loop-closure mechanisms to alleviate drifting human motion over time <ref:2604.21926#pg3>. That addresses one of those stability concerns.
Paper summary: Jane: So, it's not just about getting a static picture of movement; it's about tracking the dynamics over time and fixing those drifts as they happen <ref:2604.21926#pg3>. It’s moving from simple tracking to genuine understanding of what’s happening in a space.
Lu: The authors are pushing this idea because they believe the coupling between motion, activity, and environment is something that can be unlocked by these LLM reasoners <ref:2604.21926#pg1>. It's about using language models for cross-modal structural reasoning in a novel way <ref:2604.21926#pg3>.
Meng: I agree that the reasoning is the core idea, but we need to see how this translates into something that doesn't break at scale on a production server. The tokenization scheme itself needs to be efficient enough for high throughput <ref:2604.21926#pg3>.
Tom: So, we start with the abstract and summary of "IMU4D: 4D Human-Object Understanding from Wearable IMUs" <ref:2604.21926#pg1>. The thesis is that IMU-to-4D uses LLMs for non-visual spatiotemporal understanding to reconstruct motion and scene layouts from just a few wearable sensors <ref:2604.21926#pg1>.
Jane: And the main claim they make is that this framework yields more coherent and temporally stable results than state-of-the-art cascaded pipelines <ref:2604.21926#pg3>. They are showing that wearable motion sensors alone can support rich 4D understanding <ref:2604.21926#pg1>.
Lu: The contributions are threefold: they introduce the first framework for recovering 4D human motion, scene layouts, and textual descriptions from as few as three IMU sensors <ref:2604.21926#pg3>, they design novel tokenization schemes tailored for IMU and human motions for token-efficient yet expressive sequence modeling <ref:2604.21926#pg3>, and they show that their joint reasoning framework outperforms pipelined state-of-the-art methods <ref:2604.21926#pg3>.
Meng: The limitation they flag is that the model may struggle to distinguish between similar objects or activities when relying solely on IMU signals <ref:2604.21926#pg3>. And they also mention that it does not explicitly enforce penetration constraints, which could lead to interpenetrations in predicted three dee scenes <ref:2604.21926#pg3>.
Tom: So, the implication for us is that wearable motion sensors are a privacy-preserving, scalable alternative to vision-based perception <ref:2604.21926#pg1>. It's about building systems that perceive the world through motion rather than sight <ref:2604.21926#pg3>.
Jane: Exactly. They are showing that the coupling between motion, human activities, and environment is something that can be unlocked by these LLM reasoners <ref:2604.21926#pg1>. This opens up possibilities for ambient scene understanding without needing a camera pointed at everything <ref:2604.21926#pg3>.
Lu: The paper's main contribution is showing that joint reasoning wins over cascaded state-of-the-art modules in this area <ref:2604.21926#pg3>. It suggests that for complex 4D understanding, a unified approach is superior to piecing together many separate components <ref:2604.21926#pg3>.
Meng: I think the real-world value will come when we can deploy this kind of reasoning on actual traffic data, not just clean synthetic datasets. We need to see how robust those models are when things get messy <ref:2604.21926#pg3>.
Tom: So, we've seen the framework, the training strategy, and where they stand against prior work. The IMU4D paper is about using LLMs as spatiotemporal reasoners to turn simple inertial data into
Conclusion: Tom: So we’re wrapping up this look at IMU4D, the paper titled "IMU4D: 4D Human-Object Understanding from Wearable IMUs <ref:2604.21926#pg1>." Essentially, they show how you can understand movement and the world around you using just a few tiny sensors on your body.
Jane: Right. So we're talking about these wearable sensors—those accelerometers and gyroscopes that track motion—and they claim this framework reconstructs detailed human motion and even three dee scene layouts without needing a camera. What do you think is the big takeaway here?
Lu: The core idea is that they’re using large language models, or LLMs, not just to process data, but as these spatiotemporal reasoners. They argue that movement, what people do, and their environment are all inherently linked in a way that simple sensors can capture.
Meng: I'm trying to think about how this translates from a neat research paper into something I could actually ship. The authors say they use a joint reasoning model to link the inertial signals to human poses and scene layouts simultaneously. What does that mean for deployment costs, Lu?
Lu: It means they’re not doing separate steps for each thing. They’re using one unified formulation that models the distribution of sequential inertial signals, human motions, three dee layouts, and semantic activities all at once. It aims to infer coherent, context-aware 4D representations from just two or three sensors <ref:2604.21926#pg1>.
Jane: So it's like the model is looking at a sequence of bumps on a road and immediately figuring out not just how fast you’re going, but what kind of car you’re driving and what kind of scenery is next to you? It sounds like it’s unlocking a new way to understand context.
Tom: Exactly. And they claim this joint approach is better than the older methods that used separate pipelines. They show that their unified foundation model consistently beats those modular baselines across all motion and text evaluation metrics, which is good for me because I love seeing a clean advantage over a complex system.
Meng: But what about the real world? The paper mentions some limitations, right? What stops this from being perfect right now? I need to know what breaks when it hits actual traffic or noisy environments.
Jane: They did flag that the input is limited. They admit the model might struggle to tell two similar objects apart just by looking at the IMU signals alone, and they don't explicitly enforce penetration constraints, which could lead to predicted three dee scenes where things are inside each other.
Lu: Those are valid points for future work. The authors suggest that resolving those ambiguities would require more input signals or longer time spans in later research. They also point out that they need to incorporate loop-closure mechanisms to stop the human motion from drifting over long periods.
Tom: So, the main thing is that IMU4D shows a path toward privacy-preserving perception by relying on wearable sensors instead of cameras. It’s about building systems that perceive the world through motion rather than sight.
Jane: That’s right. The authors are demonstrating that because motion, activity, and environment are coupled, we can use LLMs to reason about them together. This opens up possibilities for sensing context without needing a camera pointed at everything.
Hao-Yu Hsu, Tianhang Cheng, Jing Wen, Alexander G. Schwing, Shenlong Wang
University of Illinois at Urbana-Champaign
cs.CV
Submitted: 2026-04-23
Updated: 2026-10-29
Comments: Project page: https://tianhang-cheng.github.io/IMU4D
Project page: https://tianhang-cheng.github.io/IMU4D
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 86/100
The gist: Understanding human activities and their surrounding environments typically relies on visual perception, yet cameras pose persistent challenges in privacy, safety, energy efficiency, and scalability.
Key concepts
- IMU-to-4D
- A framework that uses Inertial Measurement Units (IMUs) to reconstruct detailed 3D scenes and human motion. It achieves this by using Large Language Models (LLMs) to reason across sequential inertial signals, human activities, and 3D layouts simultaneously.
- Tokenization Scheme
- A novel method for converting different data types into discrete tokens suitable for the model. IMU signals are compressed using a VQ-VAE for compact root tokens, while poses use an MLP and scene layouts use one-hot labels, ensuring efficient processing of diverse inputs.
- Joint Reasoning Framework
- The core idea where LLMs are used to jointly model multiple data streams—motion, activities, 3D layouts, and text. This allows the model to infer a unified understanding of how movement relates to the surrounding environment in a single coherent representation.
- Two-Stage Training Strategy
- A training process involving two steps: first, pretraining on IMU signals for motion and text prediction, followed by finetuning on specific 3D datasets. This builds a strong foundation before adapting to different visual environments.
Terminology
Summary
Understanding human activities and their surrounding environments typically relies on visual perception, yet cameras pose persistent challenges in privacy, safety, energy efficiency, and scalability. p. 0 edu
Abstract. Understanding human activities and their surrounding environments typically relies on visual perception, yet cameras pose persistent challenges in privacy, safety, energy efficiency, and scalability. can everyday motion sensors embedded in earbuds, watches, and smartphones
reveal about our activities and environments? To answer this question, we propose IMU-to-4D, a framework for visionfree 4D perception that reconstructs detailed human motion and the surrounding
3D environment purely from inertial signals.Where the paper says it
Where the paper says it
Core Framework and Insight
The key insight behind IMU-to-4D is that Recent multimodal foundation models have
demonstrated that such coupled structure across modalities can be effectively
captured by leveraging rich pre-trained representations from LLMs and extending them to new modalities through unified architectures. Building on this intuition, IMU-to-4D repurposes large language models (LLMs) as spatiotemporal
reasoners, jointly modeling the distribution of sequential inertial signals, human
motions, 3D layouts, and semantic activities.motion, human activities, and environment are inherently coupled—everyday inertial traces encode not only how the body moves but also what surrounds it
. The framework repurposes large language models (LLMs) as spatiotemporal reasoners to jointly model the distribution of sequential inertial signals, human motions, 3D layouts, and semantic activities. p. 1Where the paper says it
coherent, context-aware 4D representations from as few as 2-3 inertial sensors
.
Tokenization and Modality Encoding
The method utilizes a novel tokenization scheme tailored for each modality to balance expressiveness, alignment, and compactness. For IMU signals, raw signals are divided into non-overlapping windows which are VQquantized using a 1D VQ-VAE with non-uniform binning to yield compact discrete root tokens. p. 5 Our
method employs a 1D VQ-VAE with non-uniform binning to tokenize root-joint motion
within fixed-length temporal windows, achieving both good reconstruction quality and
effective temporal modeling. is a sequence of Ntext discrete token indices encoding a natural language description of the observed activity. The root trajectory is
chunked into fixed-length windows and normalized, where normalized chunks are VQquantized and normalization parameters µ and σ are separately quantized via nonuniform binning, together yielding compact discrete root tokens. Body poses are continuously encoded via an MLP to produce body tokens. Text Tokens. We adopt the tokenizer and detokenizer from the pretrained
Phi-1.Where the paper says it
Where the paper says it
Where the paper says it
Training Strategy and Pretraining
The model employs a two-stage training strategy to establish a strong foundation for joint reasoning.
-
Stage 1 involves IMU-to-MotionText Pretraining, where the model is trained to jointly predict human motions and text descriptions from IMU signals using the large-scale motion-text dataset [15, 37] with synthesized IMU data. p. 7
Where the paper says it
Due to the scarcity of joint motion–text–scene datasets, we adopt a two-stage training strategy: Stage 1: IMU-to-MotionText Pretraining. Starting from Show-o’s pretrained weights, we train the model to jointly predict human motions and text descriptions from IMU signals using the large-scale motion-text dataset [15, 37] with synthesized IMU data.
-
Stage 2 is IMU-to-4D Finetuning, where the pretrained model is separately finetuned on each 3D dataset for domain adaptation to jointly optimize all modality-specific heads and the transformer backbone. p. 7
Where the paper says it
full scene layouts), we finetune the pretrained model separately on each 3D dataset [40, 57] for domain adaptation. This stage jointly optimizes all modality-specific heads, IMU encoder, and the transformer backbone.
Performance and Advantages
Experiments demonstrate that IMU-to-4D achieves – Experiments show that our joint reasoning framework outperforms pipelined
SoTA methods and enables ambient scene understanding without vision. 2 Related Work
IMU-Based Human Motion Estimation. Our architecture extends this
line of work by positioning IMU-to-4D within the multimodal foundation model
paradigm, bridging motion sensing and unified 4D understanding. 3 Method
IMU-to-4D takes sparse IMU readings and predicts 4D human motions, motion descriptions, and scene layouts.more coherent and physically consistent reconstructions than prior approaches
. The joint reasoning framework significantly outperforms pipelined SoTA methods, as shown in Table 5, where the unified foundation-model approach consistently surpasses the modular baseline across all motion and text evaluation metrics. p. 2Where the paper says it
Where the paper says it
Limitations and Future Work
Despite strong performance, certain ambiguities remain due to limited sensory input. Specifically, the model may struggle to distinguish between similar objects or activities when relying solely on IMU signals. p. 1 This unified formulation allows the model to infer coherent, context-aware
4D representations from as few as 2-3 inertial sensors—substantially improving
temporal and spatial consistency compared with cascaded pipelines (IMU →
motion → scene). Through the compositional reasoning capacity of LLMs, IMUto-4D bridges physical dynamics with spatial and semantic context, enabling
a holistic understanding of human-scene interaction without relying on visual
input. 1 Problem Formulation
Formally, our goal is to learn a model fθ that takes IMU signals Simu as input,
and outputs human motions M, motion descriptions T, and 3D scene layouts L:
M, T,L = fθ(Simu). The input IMU signal is denoted as Simu ∈ R
T ×Nimu×dimu , where T is the total
number of time frames and Nimu is the number of IMU sensors. Specifically,
we first predict the motions across all timesteps all at once. Next, conditioned
on the inputs and predicted motions, the model autoregressively decodes each
text token until an <EOT> token is generated. IMU Time Series 4D Human Scene Recovery
1. SMPL-X Motion
3.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Conclusion
IMU-to-4D introduces a framework that enables rich 4D human–scene understanding without relying on cameras by repurposing large language models for non-visual spatiotemporal reasoning. Experiments across diverse datasets show that IMU-to-4D produces more coherent and temporally stable estimates than state-of-the-art cascaded pipelines. p. 0 IMU-to-4D uses data from a few inertial sensors from earbuds, watches, or smartphones and predicts detailed
4D human motion together with coarse scene structure. Experiments
across diverse human-scene datasets show that IMU-to-4D yields more
coherent and temporally stable results than SoTA cascaded pipelines,
suggesting wearable motion sensors alone can support rich 4D understanding. We
explore an alternative: 4D perception without vision. Its goal is to reconstruct human motion and 3D scene layouts purely from everyday
wearable sensors. (2) Does large-scale pre-training on a foundation model provide advantages over task-specific specialized models? (3) Does
joint reasoning outperform a pipelined approach that cascades state-of-the-art
modules?Where the paper says it
Where the paper says it
Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Unified 4D perception without vision: The system can reconstruct detailed human motion, textual activity descriptions, and coarse 3D scene layouts purely from a few everyday wearable IMUs such as earbuds, watches, or smartphones.
-
Cross-modal reasoning via large language models: By repurposing large language models for non-visual spatiotemporal understanding, the AI can jointly model the distribution of sequential inertial signals with human motions and 3D layouts.
-
Enhanced temporal coherence through per-window normalization: The system improves reconstruction quality by normalizing poses on a per-window basis rather than using statistics from the entire dataset, which
reduces the diversity within each normalized pose sequence.
-
Robustness to sensor configuration variations: The model demonstrates robustness to different IMU device counts and placements, achieving
lower errors and more stable performance than MobilePoser (red) across different IMU configurations.
-
Superior joint reasoning over cascaded pipelines: The unified framework significantly outperforms modular baselines by learning all modalities within a single model, demonstrating the advantage of
synergistic multimodal learning over sequential pipelines.
-
3D scene re-localization: The system can infer the person's initial global translation and orientation even from sparse IMU signals by
implicitly encoding the scene structure during fine-tuning and reason[ing] about which activities are taking place and where in the scene such activities are plausible.
Abstract
Understanding human activities and the objects they interact with typically relies on visual perception, yet cameras pose persistent challenges in privacy, safety, energy efficiency, and scalability. We explore an alternative: 4D perception without vision. Its goal is to recover human motion and the objects the person interacts with purely from everyday wearable sensors. For this we introduce IMU4D, a framework that repurposes large language models for non-visual spatiotemporal understanding of human-object dynamics. IMU4D uses data from a few inertial sensors from earbuds, watches, or smartphones and predicts detailed 4D human motion together with surrounding objects the person interacts with. Experiments across diverse human motion datasets and human-object datasets show that IMU4D yields more coherent and temporally stable results than SoTA cascaded pipelines, suggesting wearable motion sensors alone can support 4D understanding of human-object interactions.
Sources
- Qwen2.5-VL Technical Report
- MotionLLM: Understanding Human Behaviors from Human Motions and Videos
- Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scaling
- Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling
- Emerging Properties in Unified Multimodal Pretraining
- SEED-X: Multimodal Models with Unified Multi-granularity Comprehension and Generation
- SnapMoGen: Human Motion Generation from Expressive Texts
- SceneMI: Motion In-betweening for Modeling Human-Scene Interactions
- OpenVLA: An Open-Source Vision-Language-Action Model
- GENMO: A GENeralist Model for Human MOtion
- Textbooks Are All You Need II: phi-1.5 technical report
- World Model on Million-Length Video And Language With Blockwise RingAttention
- HUMOTO: A 4D Dataset of Mocap Human Object Interactions
- FAST: Efficient Action Tokenization for Vision-Language-Action Models
- GenM$^3$: Generative Pretrained Multi-path Motion Model for Text Conditional Human Motion Generation
- TOTEM: TOkenized Time Series EMbeddings for General Time Series Analysis
- Emu3: Next-Token Prediction is All You Need
- Scaling Large Motion Models with Million-Level Human Motions
- VILA-U: a Unified Foundation Model Integrating Visual Understanding and Generation
- MotionStreamer: Streaming Motion Generation via Diffusion-based Autoregressive Model in Causal Latent Space
Related papers
- TEA: Text Encoder Alignment for Robust Concept Erasure in Text-to-Image Models
- Do Egocentric Video-Language Models Capture Both Hand- and Object-Centric Cues?
- Aura: Consistent Multi-Subject Video Generation via VLM-Grounded Semantic Alignment
- PVRF: All-in-one Adverse Weather Removal via Prior-modulated and Velocity-constrained Rectified Flow
- DLWM: Diverse Latent World Models for Efficient Multimodal Reasoning
- Out of Sight, Still in Mind: Token Compression for Omni-LLMs