Fine-grained Human Motion Understanding with Language Models

arXiv:2606.20888 · cs.CV · Submitted 2026-06-18 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "Fine-grained Human Motion Understanding with Language Models".

Jane: The gist In this work, FiGMo, an LLM-based model for fine-grained human motion understanding that explicitly models motion as a temporally grounded sequence of skeletal poses,

Tom: First, who's behind it and why it matters.

Paper summary: Tom: So we're looking at this paper called "Fine-grained Human Motion Understanding with Language Models" by Thomas Markhorst and his team, let's call the model FiGMo. The main idea is that understanding human motion needs to know not just *what* poses are happening, but also *when* they happen, which is a big deal for temporal stuff.

Jane: That’s right. So what this paper claims is that by representing motion as a sequence of skeletal poses with explicit timestamps for each pose, the model can reason about the action order and duration much better than before. It specifically tackles how models used to compress motion clips or just list pose features without keeping track of when they occurred makes things ambiguous when you sample time.

Lu: What I find interesting is that they use a unified pose encoder that handles both 2D and three dee skeletons, which is different from previous work that usually relied on ground-truth three dee motion capture data <ref:2606.20888#pg1,on ground-truth 3D motion capture>. They pre-train this encoder to reconstruct three dee poses from corrupted inputs using a mix of detected 2D poses, mocap data, and their projections <ref:2606.20888#pg1>.

Meng: From a deployment standpoint, I wonder how much overhead the unified encoder adds when we're trying to run this at scale? If we’re pushing this into production, we need to know if that reconstruction step is efficient enough for real-time inference on edge devices.

Lalam: Based on the literature I've reviewed, FiGMo uses a lightweight projection module called the pose translator to map those pose embeddings into the Large Language Model's embedding space. This lets the LLM combine motion, video inputs, and text prompts without constraining how we represent the motion itself.

Tom: So they’re using a multimodal LLM structure where that translator maps the features so Q can take in visual cues from videos and text prompts from language interfaces simultaneously. It sounds like they are trying to get the best of both worlds: rich temporal motion data and contextual information from video or text.

Paper summary: Jane: And then they tackle the timing issue directly by adding explicit timestamp conditioning. Instead of just feeding the pose, each pose is concatenated with its corresponding timestamp, creating prompts that look like "Time: zero point zero zero sec, Time: two point four sec, Time: three point two sec," where the token "Time" flags the timing information for the LLM to reason over <ref:2606.20888#pg2>.

Lu: That explicit timestamping is what allows them to reason about action order and duration directly from the sequence of poses and their timing tokens, which is a direct counter to models that rely on aggressive temporal subsampling which can obscure subtle motion cues <ref:2606.20888#pg3>.

Meng: But they acknowledge that existing data for this kind of pose-level grounding is limited, so what did they do about training? We know the paper claims diverse supervision is key, but what does that actually mean in practice for a team building a dataset?

Tom: They constructed their own pose-to-motion supervision by mixing different training modules. They used things like pose captioning and motion question answering alongside motion captioning and QA. The results show that the main gains come from having this diverse mix of pose-level and motion-level supervision, while the staged training approach just gives a smaller boost.

Jane: That’s important because it tells us that we don't just need one type of data to train this model; we need a variety of ways to supervise the motion understanding task so the model learns both the individual body configurations and how they change over time.

Lalam: I'm confident in this direction, as my training on similar sequence tasks suggests that having diverse supervision modules is a key driver for performance gains <ref:2606.20888#pg1>.

Tom: So to wrap up the mechanics, FiGMo uses that diverse supervision to train these pose and motion understanding capabilities, and it achieves state-of-the-art results across motion question answering, dense motion captioning, and action recognition with detected poses. They even showed that the 2D-only version surpasses previous three dee-based methods on most metrics <ref:2606.20888#pg1>.

Paper summary: Jane: It seems like the whole point of this paper is to show that by making motion representation temporally grounded with timestamps, we can achieve better timing accuracy even when dealing with things like variable frame rates or subsampling. What does this mean for us if we are trying to build applications that need to understand complex human actions in video?

Lu: It means we can move away from just classifying a static pose and start understanding the entire temporal narrative of an action, like knowing exactly how long a specific part of the squat lasted.

Meng: For me, what this implies is that if we use this approach, we might be able to deploy models that require less raw video data because they are focusing on these dense skeletal sequences instead of high-dimensional pixel inputs that are hard to handle at scale <ref:2606.20888#pg3>.

Tom: Exactly. It’s about making the representation more compact and robust to those privacy or bandwidth constraints you mentioned earlier in the abstract.

Jane: So, looking at the authors' goal, FiGMo is aiming to be a model that doesn't just see a person doing an action but truly understands the fine-grained temporal dynamics of that action by grounding every pose with its exact time.

Lu: The implication for representation learning is that we should prioritize methods like this where we explicitly encode temporal structure rather than letting the model implicitly learn it from noisy, subsampled inputs.

Meng: If you're building a system, you need to be aware that getting those precise timestamps during data collection or inference is going to be a critical part of your pipeline if you want to leverage this kind of fine-grained understanding.

Tom: So the main point here from "Fine-grained Human Motion Understanding with Language Models" is that explicit temporal grounding via timestamped skeletal poses, combined with diverse supervision, allows for reasoning about motion order and duration in a way that beats previous methods on multiple benchmarks.

Jane: And for those of us listening who aren't reading the math, it boils down to this: we can get better at understanding *when* things happen in human movement because the AI is explicitly told the timeline.

Conclusion: Tom: So we've seen how FiGMo handles motion by using timestamps on every single skeletal pose, and now we’re at the conclusion of this paper, "Fine-grained Human Motion Understanding with Language Models."

Jane: It really boils down to taking human movement and treating it like a timeline where every single frame or pose has an exact clock time attached to it.

Lu: Exactly. They built this whole system—the model, the encoder, the training—to make sure that when you ask what someone is doing, you’re not just getting a guess about their shape, but knowing exactly *when* they did each part of that movement.

Meng: So for us in production, what does that mean practically? Does it make our models faster or more stable when we're dealing with messy video feeds?

Tom: It makes the reasoning much more robust to things like variable frame rates. If you're sampling motion data, you can still get the right timing back because the model is explicitly looking at those timestamp tokens.

Jane: And that’s huge for understanding action duration. Instead of just seeing a sequence of poses, the AI now knows how long each pose lasted and what rhythm that action had.

Lu: The authors focused heavily on training it with both pose-level supervision and motion-level supervision from different datasets to make sure it learned the right way to ground those timestamps.

Lalam: My literature review confirms that this diverse supervision is what drives the main performance gains across all the tasks they tested, like question answering and captioning.

Meng: So, if I’m shipping a model, I need to know what’s the actual cost of running this full system compared to a simpler model that just looks at the frames without timing.

Tom: That’s the next big thing we have to look at—the efficiency trade-off between this high-precision temporal reasoning and what's actually deployable on real traffic.

Jane: And that brings us right back to how this changes things for people who aren't engineers—it means we can ask much more detailed questions about video, like "What was the exact timing of the pivot in their turn?"

Lu: Right. The implication is that future motion understanding won't just be about recognizing actions; it’s going to be about modeling the precise temporal dynamics of those actions.

Delft University of Technology · Honda Research Institute Japan

cs.CV

Submitted: 2026-06-18

Updated: 2026-10-15

Comments: Accepted at WACV 2027

Code: https://github.com/Thomas-Markhorst/FiGMo

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 90/100

The gist: The gist In this work, FiGMo, an LLM-based model for fine-grained human motion understanding that explicitly models motion as a temporally grounded sequence of skeletal poses, achieves

Key concepts

Timestamp-grounded skeletal poses
This means representing motion not just as a set of body positions, but as a sequence where each position is tagged with an exact time. This allows the model to know precisely *when* a specific pose occurs, which is crucial for understanding the timing and duration of actions.
Unified pose encoder
This component handles both 2D and 3D skeletal data using a single system. It is designed to be robust, capable of taking corrupted inputs (like noisy 2D poses or missing mocap data) and reconstructing accurate 3D poses, ensuring the model receives consistent motion representations regardless of the input format.
Pose-to-motion supervision
This refers to the specific training data used to teach the model how to link individual body configurations (poses) to overall motion events. By using diverse tasks like pose captioning and motion QA, the model learns not just where joints are, but how those joint changes contribute to a meaningful action.
Pose translator
This is a lightweight module that translates the specific embeddings representing skeletal poses into the mathematical space used by the main LLM. This allows the multimodal LLM to effectively combine visual information (video) and textual instructions (prompts) with motion data.

Terminology

Summary

The gist In this work, FiGMo, an LLM-based model for fine-grained human motion understanding that explicitly models motion as a temporally grounded sequence of skeletal poses, achieves state-of-the-art performance across multiple benchmarks by representing motion as a sequence of skeletal poses with explicit timestamps for each pose <ref:2606.20888#pg2>

How it works

FiGMo represents human motion as a sequence of skeletal poses with explicit timestamps for each pose, allowing the model to reason about motion order, duration, and rhythm Each pose encodes body joint positions and is temporally grounded with timestamp tokens. The model supports both 2D and 3D skeletal motion representations through a unified pose encoder. This unified encoder is pre-trained to reconstruct 3D poses from corrupted 2D or 3D pose inputs using a mixture of detected 2D poses, 3D mocap data, and their projections.

Multimodal LLM Structure

The model employs a lightweight projection module called the pose translator to map the pose embeddings into the LLM’s embedding space. This allows the multimodal LLM Q to accept optional video inputs V through its visual encoder and text prompts P through its language interface. The translated features are then passed to Q, which combines motion, video, and text information while treating video as an auxiliary cue that enhances motion understanding without constraining the motion representation space.

Interpreting Fine-grained Timing in Motion

To overcome limitations of fixed temporal resolution in previous models, FiGMo introduces explicit timestamp conditioning. Each pose is concatenated with its corresponding timestamp, producing temporally grounded prompts such as “What is this person doing? Time: 0.00 s, Time: 0.03 s, …, Time: 5.03 s ” where the token “Time” serves as a special indicator of pose timing. This explicit timestamping provides the LLM with fine-grained temporal structure, supporting reasoning about motion order, duration, and rhythm and allowing for variable frame rates.

Training with Pose- and Motion-Level Supervision

The model is trained using a diverse set of pose- and motion-level supervision modules derived from existing pose and motion datasets. These modules include pose captioning, pose QA, motion captioning, and motion QA. The training strategy involves a curriculum learning approach where different supervision modules are introduced according to input and task type, moving from pose-level supervision to motion-level supervision across four stages. Ablation studies show that the primary gains come from diverse pose- and motion-level supervision, while staged training provides a smaller additional benefit.

Evaluation and Results

FiGMo achieves state-of-the-art performance on multiple human motion understanding benchmarks, demonstrating the effectiveness of explicit timestamp grounding, unified 2D/3D pose encoding, and diverse pose- and motion-level supervision. For instance, in evaluation on BABEL-QA using 3D motion input, “Ours (3D)” achieves the best overall performance. Furthermore, when tested on CompMo dense motion captioning, FiGMo substantially outperforms UniMotion [35] and DEMO [71] across all captioning and temporal metrics The 2D-only variant of FiGMo surpasses previous 3D-based methods on most metrics.

Conclusion

FiGMo is an LLM-based method for fine-grained human motion understanding that explicitly models motion as a temporally grounded pose sequence, combining a unified 2D/3D pose encoder with explicit timestamp conditioning to enable reasoning over action order, duration, and temporally localized events. The study confirms that diverse supervision, especially pose-level data, is a key driver of downstream motion understanding. Future work could enrich timestamped motion-language reasoning with scene and object context.

Improvements for AI systems

  1. textbf Explicit Temporal Grounding in Motion Representation: The FiGMo model represents motion as a sequence of skeletal poses with explicit timestamps for each pose, allowing it to reason about motion order, duration, and rhythm. This enables the system to answer fine-grained temporal questions like What is the duration of the downward versus upward motion in the squat?

  2. textbf Unified Pose Encoder for Robust Input: The approach features a unified pose encoder that supports both 2D and 3D skeletal inputs, which learns consistent representations from mixed and partially corrupted skeletal data, improving robustness to noisy detections. This allows the model to operate effectively on practical 2D detections as well as 3D pose sequences.

  3. textbf Diverse Supervision for Enhanced Reasoning: The system employs a pose-to-motion supervision mixture spanning pose captioning, pose question answering, motion captioning, and motion question answering, showing that the primary gains come from the diversity of pose- and motion-level supervision. This enables the model to perform complex reasoning by leveraging both isolated joint positions and full sequence semantics.

  4. textbf Fine-Grained Temporal Query Answering: The explicit timestamp conditioning allows the LLM to process prompts such as What is this person doing? Time: 0.00 s , Time: 0.03 s , [etc.] This structure enables reasoning about motion order, duration, and rhythm, a capability superior to models that treat clips as atomic units.

  5. textbf Robust Training Strategy via Staged Supervision: The model utilizes a staged schedule where supervision modules are introduced sequentially from pose-level supervision to motion-level supervision, which the authors show provides improvements over single-stage training, suggesting this structure is beneficial for learning complex reasoning capabilities.

Abstract

Fine-grained motion understanding requires knowing not only which poses occur, but also when they occur. Existing motion-language models either compress motion clips or concatenate pose features without retaining the timing of each pose, making action duration and temporal scale ambiguous under temporal subsampling. We introduce FiGMo, an LLM-based model that represents motion as timestamp-grounded skeletal poses, supporting reasoning over action order and duration. A unified pose encoder applies this representation to both 2D and 3D skeletons, including poses detected from real videos, while RGB can optionally provide contextual information. To reason over timestamped pose sequences, the model must understand both individual body configurations and how they change over time. As existing data provides limited pose-level grounding, we construct the required pose-to-motion supervision. Controlled experiments show that timestamps improve temporal localization and preserve physical timing under subsampling, while the constructed supervision drives the main performance gains. FiGMo achieves state-of-the-art results across motion question answering, dense motion captioning, and action recognition with detected poses. Our code is available at: https://github.com/Thomas-Markhorst/FiGMo

Sources

Related papers