AnyMo: Geometry-Aware Setup-Agnostic Modeling of Human Motion in the Wild

summary

The gist

AnyMo introduces a geometry-aware framework for setup-agnostic human motion modeling in the wild As wearable and mobile devices become increasingly embedded in daily life…

In short

AnyMo is a geometry-aware framework for modeling human motion using wearable sensors regardless of setup. It simulates realistic sensor placements on the body, learns setup-agnostic representations from these simulations, and aligns them with language. This allows the model to recognize activities and generate descriptions from sparse wearable data even when tested on new subjects or devices.

Key concepts

Geometry-Aware Simulation
This involves using physics equations to simulate where an IMU sensor would realistically be placed on a body surface. It creates virtual sensor frames based on anatomical axes and surface normals, generating diverse, plausible IMU signals that mimic real-world wearable setups.
Setup-Agnostic Representation Learning
The model learns a representation of motion from simulated data organized in a graph structure. This learning treats variations in sensor placement and mounting orientation as part of the setup variation, enabling the encoder to capture the underlying temporal dynamics and cross-segment motion correlations.
Motion-Language Modeling
This process involves converting IMU data into discrete tokens and training a language model to understand them. This teaches the model to align specific wearable motion tokens with natural language descriptions of activities, improving zero-shot recognition and captioning capabilities.
Multi-task Contrastive Instruction Tuning
This technique uses a shared large language model backbone to simultaneously align IMU prompts with various tasks like activity labeling, cross-modal retrieval, and captioning. This joint training strengthens the model's ability to generalize across different types of motion understanding problems.

Terminology used across episodes

This episode discusses

The paper

AnyMo: Geometry-Aware Setup-Agnostic Modeling of Human Motion in the Wild · Read on arXiv

Baiyu Chen, Zechen Li, Wilson Wongso, Lihuan Li, Xiachong Lin, Hao Xue, Benjamin Tag1, Flora Salim

University of New South Wales · Hong Kong University of Science and Technology (Guangzhou) · Hong Kong University of Science and Technology

As wearable and mobile devices become increasingly embedded in daily life, they offer a practical way to continuously sense human motion in the wild. But inertial signals are highly dependent on the sensing setup, including body location, mounting position, sensor orientation, device hardware, and sampling protocol. This setup dependence makes it difficult to learn motion representations that transfer across devices and datasets, and limits the broader use of wearable IMUs beyond closed-set recognition. We introduce AnyMo, a geometry-aware framework for setup-agnostic human motion modeling. AnyMo uses physics-grounded IMU simulation over dense body-surface placements to generate diverse and plausible synthetic signals, pre-trains a graph encoder from paired synthetic placement views and masked partial observations, tokenizes multi-position IMU into full-body motion tokens, and aligns these tokens with an LLM for motion-language understanding. We evaluate AnyMo on three complementary tasks: zero-shot activity recognition across 14 unseen downstream datasets, cross-modal retrieval, and wearable IMU motion captioning, where it improves average Accuracy/F1/R@2 by 11.7%/11.6%/22.6% on HAR, increases zero-shot IMU-to-text and text-to-IMU retrieval MRR by 15.9% and 28.6%, respectively, and improves zero-shot captioning BERT-F1 by 7.5%. These results support AnyMo as a generalist model for wearable motion understanding in the wild. Project page: https://anymo-project.github.io.

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "AnyMo: Geometry-Aware Setup-Agnostic Modeling of Human Motion in the Wild".

Jane: Here's the gist of the paper. AnyMo introduces a geometry-aware framework for setup-agnostic human motion modeling in the wild As wearable and mobile devices become increasingly embedded in daily life, they offer a practical way to continuously sense human motion in the wild This paper presents AnyMo, a geometry-aware framework for setup-agnostic human motion modeling.

Tom: The Core Problem Addressed Inertial signals are highly dependent on the sensing setup, including body location, mounting position, sensor orientation, device hardware, and sampling protocol This setup dependence makes it difficult to learn motion representations that transfer across devices and datasets The paper identifies three coupled axes that make building a broadly useful wearable motion model difficult: Data and Supervision Scarcity, Limited Realism of Synthetic Augmentation, and Modality Gap These challenges motivate the key insight that wearable setup variation is structured rather than arbitrary.

Jane: The AnyMo Pipeline AnyMo connects physics-grounded geometry-aware IMU simulation with geometry-aware setup-agnostic pre-training, full-body IMU tokenization, and motion-language alignment The pipeline involves three key enablers: geometry-aware IMU simulation (Section 3.1), setup-agnostic representation learning (Section 3.2), and full-body IMU tokenization with motion-language alignment (Section 3.3).

Tom: Geometry-Aware Simulation The physics engine synthesizes wearable IMU windows by applying wearable IMU motion equations to synchronized Nymeria body motion to model plausible wearable locations on the body surface For each anatomical segment i, the local sensor frame is defined using an anatomical axis ui and a surface normal ni,v to form a right-handed surface-based sensor frame Rsurf i,v.

Title and authors: Tom: So we're looking at this paper called "AnyMo: Geometry-Aware Setup-Agnostic Modeling of Human Motion in the Wild". Jane, what's the main thing that catches your eye about the title and who wrote it?

Jane: Well, it sounds a bit technical. It’s about building a model for human motion that doesn't care where you put the sensor or what device you use. That’s pretty big for wearable stuff, right?

Lu: Exactly. The paper tackles the problem that inertial signals are super dependent on the sensing setup, like where your body is, how you’re holding your phone, and what kind of sensor it is. This makes learning motion representations that work across different devices really hard.

Meng: So there’s this setup dependence issue they want to fix. But usually, we just train a model on one setup—say, a wrist watch—and then it fails when you try to use a phone or glasses because the sensor placement is completely different.

Tom: Right. And the authors say that wearable setup variation isn't random; it has this structure, which is what they call three coupled axes that make building these models difficult. It’s not just one thing going wrong.

Jane: So they break down the problem into data and supervision scarcity, limited realism in synthetic training, and a modality gap between the sensor data and language descriptions. That sounds like a solid breakdown for a complex issue.

Lu: They suggest that we need to address these three things by connecting physics-grounded simulation with setup-agnostic pre-training, full-body IMU tokenization, and then motion language alignment. That’s the pipeline they propose to solve it.

Meng: Physics simulation sounds cool, but how realistic do you think these synthetic signals are going to be compared to real data? Because if the simulation is too far off, all the pre-training won't help when you move to the real world.

Tom: That’s a fair question. The paper claims they use physics-grounded IMU simulation over dense body-surface placements to generate diverse and plausible synthetic signals, which they say bridges that gap between synthetic pre-training and real wearable IMU data.

Jane: So imagine you have this simulation that generates many different plausible locations for a sensor on your body, which is a big step because it gives the model more variety than just one fixed setup.

Lu: They define a right-handed surface-based sensor frame for each anatomical segment using an anatomical axis and a surface normal, which helps them model these virtual IMU trajectories based on synchronized body motion. That’s how they make the simulation geometry aware.

Title and authors: Meng: Geometry awareness is important, but what about the learning part? How does the setup-agnostic representation learning actually learn to ignore where you are mounting it?

Lu: They organize these simulated IMU candidates into a spatio-temporal graph where each node stores an IMU window sampled for a specific body segment, and they use a graph encoder to model temporal dynamics while treating surface placement and mounting orientation as within-segment setup variation.

Tom: So the encoder is learning how motion flows across different body parts even when the physical sensor setup might be slightly different on that part. That’s where I get excited about the mechanics of it.

Jane: And they use a masked cross-view predictive contrastive learning objective, which teaches the encoder to infer full-body motion from sparse wearable inputs by contrasting it against other views, which helps separate different motion instances.

Meng: Okay, so they are essentially training the model to be robust to setup changes by making it predict the whole body when you only see a tiny bit of it in one specific spot. That sounds like a lot of data manipulation for pre-training.

Tom: The results show that removing geometry-aware simulation causes the largest degradation in performance, which tells us that getting those realistic surface-aware setup variations is crucial for the model to be useful outside of a controlled lab setting.

Jane: It seems like they’ve made a lot of noise reduction by including both the realistic simulation and that contrastive learning objective. But how do they actually turn all this complex motion data into something useful for an application, like activity recognition?

Lu: They take the frozen graph encoder latent and discretize it into compact IMU tokens using a product quantizer, which results in discrete token sequences that preserve the temporal order of the setup-stable motion latent.

Meng: So they are compressing all that complex spatial and temporal information down into a vocabulary of these tokens. That’s what makes it manageable for later stages, but I wonder how much detail they lose in that quantization step.

Tom: They then use these discrete tokens in a motion language model pre-training, where they replace raw IMU token embeddings with projected codebook vectors to introduce the IMU vocabulary into the large language model.

Jane: That means the LLM starts understanding these specific wearable motion tokens instead of just general text, which is supposed to help it generate captions that better preserve action-level semantics.

Lu: And they use a multi-task contrastive instruction tuning approach, which jointly supports embedding-based and generation-based tasks to align those IMU prompts with language descriptions for zero-shot recognition, cross-modal retrieval, and motion captioning.

Title and authors: Meng: So they are using the same LLM backbone for different jobs—recognition and captioning—by just changing the input prompt structure, which is smart if you want efficiency at scale.

Tom: The evaluation is pretty comprehensive because they test it on three complementary tasks: zero-shot activity recognition across fourteen unseen datasets, cross-modal retrieval on Nymeria held-out data, and wearable IMU motion captioning.

Jane: And the numbers are pretty encouraging; for zero-shot recognition, AnyMo gets a thirty-five point seven accuracy, twenty-nine point five macro F1, and fifty-seven point five recall at two. That’s a solid performance across all those metrics.

Lu: They also show substantial outperformance in cross-modal retrieval on Nymeria held-out data compared to all the baselines they tested on that unseen set.

Meng: For deployment purposes, I’d want to know how much computational cost this whole pipeline adds when you run it on a real device, especially with that simulation step involved. Does it run fast enough for production?

Tom: The ablation study confirms the necessity of key components; removing the geometry-aware simulation causes the largest degradation in performance, and replacing that contrastive objective also severely hurts performance, highlighting how important both are.

Jane: It seems like they’ve proven that combining geometry-aware wearable simulation with motion language modeling is a promising path toward generalist motion understanding from sparse wearable sensors.

Lu: The paper suggests that this combination of physics-grounded simulation and motion language modeling forms more discriminative activity clusters while aligning real and synthetic samples of the same activity.

Meng: So, for someone who just listens to the show, what does this actually mean? Does it mean we can finally build wearable models that work reliably in the wild without needing a perfect setup beforehand?

Tom: It means we’re moving toward a model that can understand human motion from sparse wearable sensors even when those sensors are mounted in unexpected ways.

Jane: It’s about making these models generalist tools for understanding motion outside of controlled environments, which is really practical for everyday use.

Lu: The authors suggest that combining geometry-aware wearable simulation with motion language modeling is a promising path toward generalist motion understanding from sparse wearable sensors because it addresses the setup dependence issue structurally.

Meng: It sounds like they've made progress in making these models more robust to real-world sensor variability, which is something I’m really interested in for deployment challenges.

Tom: So, that’s AnyMo: Geometry-Aware Setup-Agnostic Modeling of Human Motion in the Wild. That’s where we are on this paper.

The paper's summary: Tom: So, we've been looking at AnyMo, which is this new framework for modeling human motion that tries to work regardless of the setup you use or what device you have on your body.

Jane: That’s right, it’s about making wearable motion models setup-agnostic. Basically, they built a system that learns how people move from sparse sensor data even when that sensor is mounted somewhere completely different than it was during training.

Lu: The core idea they are pushing is connecting physics simulation with representation learning and then language modeling to make this generalist model work. They tackle the problem of setup dependence by showing that wearable variation isn't random, it has a structure they can exploit.

Meng: So, instead of just training on one way you hold your watch or phone, they are using geometry-aware simulation to create many different plausible ways a sensor could be placed on the body. That gives them this huge variety of training data candidates.

Tom: Exactly. The simulation creates these virtual IMU trajectories based on the actual physics of how that sensor would move relative to your body's movement, which is way more realistic than just randomly moving a sensor around in a computer model.

Jane: And then they feed all those different simulated setups into this graph encoder, which learns the underlying motion patterns across different body segments while ignoring the specific mounting details. That’s how they get that setup-agnostic representation.

Lu: They use something called masked cross-view predictive contrastive learning to train that encoder, which essentially forces it to learn how to predict the whole body motion from just a small piece of a wearable sensor data, contrasting it against other simulated views.

Tom: That's a big mechanical detail. It’s like teaching the model: "If you see this tiny bit of movement here, predict where everything else is going based on what we know about human motion."

Jane: And then they take those learned representations and turn them into discrete tokens using a product quantizer, which keeps the temporal order of the motion structure intact even when compressing it down.

Lu: These tokens are then plugged into a language model, where they replace standard embeddings so the large language model actually learns what these specific wearable motion tokens mean in context.

Meng: So you’re essentially building a specialized vocabulary for human movement that is tied to physics and geometry, rather than just abstract data points.

Tom: And they tie it all together with multi-task contrastive instruction tuning, which aligns those motion tokens directly with language descriptions for tasks like activity recognition and captioning.

Jane: The results show this works pretty well across the board: zero-shot recognition on fourteen different datasets, cross-modal retrieval on unseen data, and even captioning transfer to a completely different dataset.

Lu: They've shown that removing the geometry simulation hurts performance the most, confirming that modeling realistic surface variation is what unlocks this generalization.

Tom: And they also showed that the contrastive learning objective is vital; without it, the model doesn't learn to separate those different motion instances well enough for open-vocabulary recognition.

Jane: It really suggests that combining physics-grounded simulation with this kind of representation learning and language alignment is a solid way forward for building wearable motion understanding that actually works in the real world.

Lu: They conclude by showing that this approach creates more discriminative activity clusters while making sure the synthetic training samples match the real ones, which is a pretty key finding.

Meng: For me, what matters most is that they managed to create something robust enough to handle the variability we see when people use these devices in their actual lives, not just in a controlled lab setting.

Tom: Exactly. It’s about moving past models that only work perfectly when the sensor is exactly where you put it on the body. AnyMo aims for that real-world applicability.

Jane: So, AnyMo is presenting a path toward generalist motion understanding from sparse wearable sensors by tackling setup dependence with physics simulation and language modeling.

Lu: The implication is that we can start building systems that understand human activity from just a few points of sensor data, even if the mounting position changes every time.

The paper's improvements: Tom: So, we’re looking at how AnyMo suggests fixing its own issues to make it even better than what we just saw. It’s basically an ablation study showing exactly which parts of the method are necessary for it to perform well in a real scenario.

Jane: That’s right, they show that if you take out the geometry-aware simulation, performance drops significantly. That tells us that getting those realistic surface-aware setup variations is crucial for any generalization we want.

Lu: They also showed that removing the masked cross-view predictive contrastive objective hurts performance badly too. It confirms that training the encoder to infer full body motion from sparse inputs by contrasting them against other views is really important for separating different motion instances.

Tom: That’s a clean win for me. It proves that both realistic setup variation and this specific way of pre-training are necessary to get that model working outside of a super controlled environment.

Jane: And they pointed out that label-level supervision and free-form narration alignment both help with zero-shot recognition, which is interesting because it means you don't always need perfect human labels to get good results.

Lu: They also showed that relying only on next-token instruction tuning isn't enough for open vocabulary recognition; you need that contrastive tuning to make the model discriminative.

Tom: So, the numbers confirm what we suspected: if you remove these key components, the performance tanks. It’s a very clear indicator of where they need to focus their development effort next.

Jane: What this means for us is that if we want to build something truly generalist for wearable motion, we can't just rely on one clever trick. We have to combine the physics simulation with this specific contrastive training objective and the language alignment.

Lu: The paper suggests that combining geometry-aware simulation with motion language modeling creates more distinct activity clusters while simultaneously aligning real and synthetic samples of the same activity. That’s a solid result for consistency.

Tom: So, they aren't just building a better recognition tool; they’re making sure the synthetic data matches the real data structure in a way that actually helps the model learn.

Jane: It really moves us closer to having wearable models that are reliable even when someone wears them in an unexpected way or uses them on a device we haven't seen before.

Lu: The authors suggest this combination of physics simulation and language modeling is a promising path toward generalist motion understanding from sparse wearable sensors because it addresses the setup dependence issue structurally.

Tom: It sounds like the next step is to focus on making sure those tokens are really robust and that the alignment between language and motion stays tight across all those tasks.

Jane: It’s about building a system that understands human movement in the wild, regardless of where or how you're sensing it.

Conclusion: Tom: So, to wrap up AnyMo, we’ve seen how they use geometry-aware simulation to build this setup-agnostic model for human motion in the wild.

Jane: That means they’ve created a way for AI to understand how people move from just a few sensor readings, no matter where those sensors are mounted.

Lu: It boils down to connecting physics simulation with representation learning and language modeling to make this generalist model work across different devices and setups.

Meng: From an engineering standpoint, the fact that they removed the geometry simulation showed us that getting that realistic surface variation is a major performance driver; it’s not just adding complexity for complexity' sake.

Tom: Exactly, and they also proved that this specific contrastive training objective is necessary to make sure the model can actually separate different motion instances effectively.

Jane: It’s about making these models robust so they don't break when you move them from the lab to a real-world situation where the sensor placement is completely unknown.

Lu: The conclusion is that combining physics-grounded simulation with motion language modeling forms more distinct activity clusters while matching real and synthetic samples of the same activity.

Meng: So, for deployment, this suggests we can build tools that are reliable even when they’re not perfectly calibrated in a known environment.

Tom: It’s about moving toward a model that actually understands human motion from sparse wearable sensors without needing a perfect setup beforehand.

Jane: AnyMo presents a promising path toward generalist motion understanding from sparse wearable sensors by addressing the setup dependence issue structurally.

Lu: They’ve shown that this approach creates more discriminative activity clusters while aligning real and synthetic samples of the same activity, which is a pretty key finding for representation learning.

Meng: I just wonder how much computational overhead this whole pipeline adds when you run it on a real device with all that simulation involved.

Tom: That’s a fair question, Meng; we need to see if this complexity translates into something practical for production use.

Jane: It sounds like the next big challenge is making sure this level of setup-agnostic learning can scale up effectively without requiring massive amounts of new labeled data for every single device.

Lu: I think the literature shows that connecting geometry-aware simulation with motion language modeling is a solid way forward, but we still need to explore how to make that tokenization step even more efficient.

Meng: The paper doesn't spend a lot of time on the long-term scaling issues for deployment, which is something I’m always worried about when I look at these kinds of complex systems.

Tom: Still, AnyMo gives us a really strong foundation for understanding how to model human motion in the wild.

Jane: It’s about making these models generalist tools for understanding motion outside of controlled environments, which is really practical for everyday use.

More episodes

← Home