Linear Multi-granular Body Modeling for Text-Driven Motion Generation

arXiv:2503.06897 · cs.CV · Submitted 2025-03-10 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "Linear Multi-granular Body Modeling for Text-Driven Motion Generation".

Jane: The gist The proposed HiSTF Mamba framework introduces a novel hierarchical spatiotemporal fusion approach that combines local and global spatial features with bidirectional temporal modeling to enhance text-to-motion generation.

Tom: First, who's behind it and why it matters.

Title and authors: Tom: So, let's start by looking at what this paper is calling Linear Multi-granular Body Modeling for Text-Driven Motion Generation and who came up with it. It’s a framework designed to handle motion generation driven by text prompts.

Jane: The authors are Xingzu Zhan, Chen Xie, Honghang Chen, Haoran Sun, and Xiaochun Mai from Shenzhen University. They’ve put together this HiSTF Mamba framework to address those issues we just mentioned.

Lu: It’s interesting because they are taking the concept of multi-granular modeling and adding a new spatiotemporal fusion strategy using a Mamba architecture to improve efficiency.

Meng: So, what does this actually mean for someone trying to build a system that generates realistic human motion from text? What is the immediate value here?

Lalam: It means we can generate motions that are not just coordinated globally but also have those specific, small movements happening at the joints.

The paper's summary: Tom: The HiSTF Mamba framework is built around three main parts: the Dual-Spatial Mamba, the Bi-Temporal Mamba, and then that Dynamic Spatiotemporal Fusion Module or DSFM. It’s a hierarchical approach.

Jane: That structure is key because it tackles the problem by separating how it handles spatial features from how it handles time dependencies, and then fusing them smartly at the end.

Lu: The Dual-Spatial Mamba uses a part-based and whole-based learning strategy simultaneously to make sure we capture both the large movements and the fine joint dynamics.

Meng: So, they are trying to learn everything at once—the big picture coordination and the tiny details of individual joints—in one go. That sounds computationally demanding.

Lalam: It’s designed to ensure that when it learns about the whole body movement, it doesn't forget the small movements happening in a specific limb or joint.

The paper's improvements: Tom: Moving onto what they actually improved, they introduced several specific enhancements. First, the Bi-Temporal Mamba block uses a bidirectional scanning mechanism to look at motion sequences both forward and backward.

Jane: This is supposed to let the model capture both the short-term motion details and those longer dependencies in a much better way than before.

Lu: And then they have this Dynamic Spatiotemporal Fusion Module, or DSFM, which specifically removes redundancy from the temporal features before fusing them with the spatial features.

Meng: So, how does that fusion actually translate to better results when you’re generating motion? Is it just a minor tweak in accuracy?

Lalam: The authors claim this significantly enhances the realism and smoothness of the generated motion sequences by extracting complementary information from those temporal features.

Conclusion: Tom: So, to wrap up on Linear Multi-granular Body Modeling for Text-Driven Motion Generation, this framework is about combining local spatial details with bidirectional temporal modeling to make better text-to-motion.

Jane: It validates that by using the Dual-Spatial Mamba and the DSFM, we can get a representation of motion that has both high fidelity and strong text relevance.

Lu: The key finding they highlighted is a significant improvement in how well it balances those two things, which is what they call high fidelity and strong text relevance.

Meng: From an engineering standpoint, the efficiency gains from using Mamba over some other methods are also worth mentioning, even though the paper focuses more on quality.

Lalam: It’s promising because it means we can expect motions that are much more detailed at the joint level compared to what we saw in earlier models like MotionMamba before this work.

Tom: So, that’s a solid summary of Linear Multi-granular Body Modeling for Text-Driven Motion Generation. We’ll be looking at how these ideas play out in other papers next.

Shenzhen University

cs.CV

Submitted: 2025-03-10

Updated: 2026-10-09

Comments: Accepted by ICME 2026

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 79/100

The gist: The gist The proposed HiSTF Mamba framework introduces a novel hierarchical spatiotemporal fusion approach that combines local and global spatial features with bidirectional temporal modeling to

Key concepts

Dual-Spatial Mamba: This module performs parallel learning by modeling motion at two levels simultaneously: part-based and whole-based. It decomposes the motion into individual body parts to learn fine joint dynamics, while also retaining the entire motion matrix input to capture global coordination. These two distinct learning strategies are then combined for a comprehensive representation.
Bi-Temporal Mamba: This block uses a bidirectional scanning mechanism within Selected SSMs. This allows the model to process motion sequences in both forward and backward directions, enabling it to effectively encode both short-term motion details and long-term dependencies within the sequence, leading to more coherent movements.
Dynamic Spatiotemporal Fusion Module (DSFM): The DSFM module refines temporal features by removing redundancy and extracting complementary information before integrating them with spatial features. It uses an attention matrix approach to fuse these refined temporal features with spatial data via concatenation and linear projection, resulting in a more expressive spatiotemporal representation.

Terminology

Alignment — 1× in this paper
Benchmark — 1× in this paper · explained in 2 episodes
Dynamic Spatiotemporal Fusion Module — 1× in this paper
Fidelity — 1× in this paper · explained in 2 episodes
Multimodal Learning — 1× in this paper · explained in 2 episodes
Redundancy — 1× in this paper · explained in 2 episodes
Semantic Alignment — 1× in this paper · explained in 4 episodes

Summary

The gist The proposed HiSTF Mamba framework introduces a novel hierarchical spatiotemporal fusion approach that combines local and global spatial features with bidirectional temporal modeling to enhance text-to-motion generation.

How it works

HiSTF Mamba is composed of three key modules: Dual-Spatial Mamba, Bi-Temporal Mamba, and the Dynamic Spatiotemporal Fusion Module (DSFM) The Dual-Spatial Mamba incorporates “part-based + wholebased” parallel modeling to capture both whole-body coordination and fine-grained joint dynamics. This strategy ensures the model can capture fine-grained joint details while preserving coarse-grained whole-body coordination.

Bi-Temporal Mamba

The Bi-Temporal Mamba block employs a bidirectional scanning mechanism that effectively encodes short-term motion details and long-term dependencies. It utilizes Selected SSMs featuring a bidirectional scanning mechanism to capture the contextual information of motion sequences in both forward and backward directions, thereby generating more coordinated and coherent motion sequences. This bidirectional strategy is particularly effective for modeling motion sequences.

Dual-Spatial Mamba

This module performs two types of learning at the same time through Multi-Granular Body-Spatial Modelling. It adopts a parallel Part-based and Whole-based learning strategy. The Part-based phase decomposes the entire motion matrix into 6 body part matrices, allowing the model to focus on learning body part joint details. The Whole-based phase retains the entire motion matrix inputs to learn whole-body motion coordination. These two features are then fused via an element-wise addition strategy to obtain a representation that combines both global motion patterns and fine-grained joint-level detail features.

Dynamic Spatiotemporal Fusion Module (DSFM)

The DSFM module is designed to perform redundancy removal and extraction of complementary information for temporal features. It modifies the fusion strategy to effectively integrate these refined temporal features with spatial features. Specifically, it first stacks the N-1 temporal features and splits them into a query Yq and a key Yk for computing an attention matrix. The resulting refined temporal features are then integrated with the spatial features by concatenating them along the feature dimension before applying a linear projection to map back to R(T ×D).

Experimental Results

Extensive experiments on HumanML3D demonstrate that HiSTF Mamba achieves outstanding results across multiple metrics. Notably, it reduces the FID score from 0.281 to 0.189, a reduction of nearly 30%, while also achieving leading performance on other metrics. The model produces motions that more closely align with the text, exhibiting more detailed joint movements and better full-body coordination. Ablation studies confirm the effectiveness of all proposed components in capturing both fine-grained joint details and overall body coordination. The model maintains a parameter count of just 15.53 M, around 13% fewer than the baseline MDM.

Conclusion

HiSTF Mamba introduces a text-to-motion generation framework that combines local and global spatial features with bidirectional temporal modeling. Its Dual-Spatial Mamba block captures both fine-grained joint dynamics and coarse whole-body movements, while the Bi-Temporal Mamba block encodes short-term and long-term dependencies in a bidirectional manner. To fuse refined temporal features with spatial representations, it proposes the DSFM block, which reduces redundancy and boosts overall spatiotemporal expressiveness. Extensive experiments on HumanML3D and KIT-ML confirm that HiSTF Mamba achieves robust performance across multiple evaluation metrics while producing high-fidelity and semantically aligned motion sequences. This method provides a promising avenue for generating realistic, detail-rich human motions. The model achieves these results with efficient training and rapid inference.

--- Page 1 ---

Conclusion

HiSTF Mamba introduces a text-to-motion generation framework that combines local and global spatial features with bidirectional temporal modeling.

Improvements for AI systems

Dual-Spatial Mamba for Part-Based Modeling captures fine-grained joint details by decomposing the motion matrix into 6 body part matrices, allowing the model to focus on local kinematics while simultaneously retaining whole-body coordination. This enables generation of motions that exhibit both full-body coordination and joint-level detail.
Bi-Temporal Mamba utilizes a bidirectional scanning strategy, which is described as capturing short-term cues and long-term dependencies to ensure temporal coordination, leading to the generation of more coordinated and smooth motion sequences.
Dynamic Spatiotemporal Fusion Module (DSFM) performs crucial feature refinement by removing redundancy from temporal features and then fusing them with spatial features, yielding a representation that preserves both joint-level detail and overall body coordination.
The HiSTF Mamba framework achieves robust performance across multiple evaluation metrics, specifically reducing the FID score from 0.281 to 0.189 on HumanML3D, validating its effectiveness in achieving high fidelity and strong semantic alignment in text-to-motion generation.