Less is More: Compact-Token Masked Feature Prediction for Skeleton Representation Learning
summary
The gist
Less is More: Compact-Token Masked Feature Prediction for Skeleton Representation Learning proposes SLiM as a compact-token framework that unifies masked feature prediction and contrastive learning…
In short
SLiM proposes a compact-token framework to unify masked feature prediction and contrastive learning for skeleton representation. It shifts the focus from dense coordinate reconstruction to teacher-guided semantic feature prediction using a small token grid, significantly reducing inference computation while maintaining state-of-the-art performance.
Key concepts
- Compact Token Grid
- Instead of using dense sequences of tokens, SLiM employs a much smaller grid (e.g., 8x25 tokens) to represent skeletal data. This compact structure is key to drastically lowering the computational cost during the prediction phase compared to traditional methods.
- Masked Feature Prediction (MFP)
- This involves training a model to predict features of masked parts of a skeleton sequence, guided by a teacher network's output. This replaces raw coordinate reconstruction, allowing the model to learn high-level semantics instead of just precise joint positions.
- Semantic Tube Masking (STM)
- To prevent shortcuts from overly dependent joints, STM occludes connected anatomical regions across consecutive frames. This forces the model to perform deeper skeletal-temporal reasoning and ensures anatomical consistency during training.
Terminology used across episodes
This episode discusses
- Less is More: Compact-Token Masked Feature Prediction for Skeleton Representation Learning · Paper Radio
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- HYperbolic Self-Paced Learning for Self-Supervised Skeleton-based Action Representations
- Part Aware Contrastive Learning for Self-Supervised Action Recognition
- PKU-MMD: A Large Scale Benchmark for Continuous Multi-Modal Human Action Understanding
- SGDR: Stochastic Gradient Descent with Warm Restarts
- Decoupled Weight Decay Regularization
- DINOv2: Learning Robust Visual Features without Supervision
- iBOT: Image BERT Pre-Training with Online Tokenizer
The paper
Less is More: Compact-Token Masked Feature Prediction for Skeleton Representation Learning · Read on arXiv
Information & Electronics Research Institute of KAIST
Current skeleton representation learning paradigms face distinct limitations: Contrastive Learning (CL) often overlooks fine-grained motion details, while Masked Auto-Encoders (MAE) rely on coordinate-level reconstruction. This reconstruction inherently demands dense token sequences and heavy decoders, wasting pre-training computation on discarded components and forcing downstream inference to process dense token grids. To resolve these bottlenecks, we propose SLiM (Skeleton Less is More), a compact-token framework that unifies masked feature prediction and contrastive learning via a shared encoder. By shifting the objective from raw coordinate reconstruction to decoder-free, teacher-guided feature prediction, SLiM breaks the reliance on dense tokenization and enables effective learning with a highly compact token grid. Crucially, to prevent trivial shortcut learning arising from strong inter-joint dependencies of human, we introduce Semantic Tube Masking together with Skeleton-Aware Augmentations to enforce deep skeletal-temporal reasoning and anatomical consistency. Extensive experiments across multiple downstream protocols demonstrate that SLiM achieves state-of-the-art performance while structurally reducing inference computation by 7.89 times compared to dense-token MAE baselines.
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: We're looking at the paper: "Less is More".
Jane: Here's the gist of the paper. Less is More: Compact-Token Masked Feature Prediction for Skeleton Representation Learning proposes SLiM as a compact-token framework that unifies masked feature prediction and contrastive learning via a shared encoder The authors propose SLiM, a compact-token framework that unifies masked feature prediction and contrastive learning via a shared encoder By shifting the objective from raw coordinate reconstruction to decoder-free, teacher-guided feature prediction, SLiM breaks the reliance on dense tokenization and enables effective learning with a highly compact token grid Crucially, to prevent trivial shortcut learning arising from strong inter-joint dependencies of human, they introduce Semantic Tube Masking together with Skeleton-Aware Augmentations to enforce deep skeletal-temporal reasoning and anatomical consistency.
Tom: Extensive experiments across multiple downstream protocols demonstrate that SLiM achieves state-of-the-art performance while structurally reducing inference computation by 7.89× compared to dense-token MAE baselines.
Jane: Pruning redundant reasoning tokens Current skeleton representation learning paradigms face distinct limitations: Contrastive Learning (CL) often overlooks fine-grained motion details, while Masked Auto-Encoders (MAE) rely on coordinate-level reconstruction.
Tom: This reconstruction inherently demands dense token sequences and heavy decoders, wasting pre-training computation on discarded components and forcing downstream inference to process dense token grids.
Paper summary: Tom: So, looking at "Less is More: Compact-Token Masked Feature Prediction for Skeleton Representation Learning" and what we’ve seen, it boils down to using a decoder-free teacher-guided feature prediction objective combined with contrastive learning to learn skeleton representations.
Jane: The core idea seems to be that by targeting high level semantics instead of low level coordinates, you can break the dependency on dense tokenization, which is what really makes these current methods heavy.
Lu: It’s about using a compact token grid for the encoder and adding specific masking strategies like Semantic Tube Masking to keep that compact representation accurate enough for downstream tasks.
Meng: From an engineering standpoint, the seven point eight nine times reduction in inference cost is the big win because it shows a clear path to deploying these models on devices where computation is limited.
Tom: And they are showing that this compact token approach still achieves state of the art accuracy on several protocols while being structurally much lighter than dense token MAE baselines.
Jane: So, for someone listening who doesn't read the math, it means we can potentially get better action understanding systems that use less power and run faster without losing too much accuracy.
Conclusion: Tom: So, we're wrapping up this look at "Less is More: Compact-Token Masked Feature Prediction for Skeleton Representation Learning." Essentially, these authors are proposing a way to learn how skeletons move using much less data and less computation than what’s currently standard.
Jane: That sounds really simple on the surface. So, what is the main trick they used to make it work with fewer tokens?
Lu: The main trick is moving away from trying to reconstruct every single joint coordinate directly. Instead, they use this shared encoder—a compact token grid—to predict high-level features instead of low-level coordinates.
Meng: Because they aren't reconstructing dense points, it means the whole setup is much leaner computationally, which is what I care about when we’re trying to deploy models on real hardware.
Tom: Right, and they did this by combining two things: masked feature prediction and contrastive learning all through one shared encoder. That’s the framework they built.
Jane: But if it’s just a compact grid, won't that lose the fine motion details we need? What stops them from just getting blurry results?
Lu: They actually added specific techniques to stop that. They introduced Semantic Tube Masking and Skeleton-Aware Augmentations to make sure the model learns deep temporal reasoning and anatomical consistency instead of just memorizing a few pixels.
Meng: I’ve seen those kinds of augmentation tricks before, but if it helps keep the representation robust under real-world noise, that makes sense for production readiness.
Tom: The results they show are pretty strong too. They managed to beat some existing methods on several tasks while also cutting the inference cost by almost eight times compared to those dense token methods.
Jane: So, what does this mean practically? For someone who just listens to the show, what’s the real-world impact of learning skeletons with less data and less power?
Lu: It suggests that you can build action understanding systems for things like robotics or rehabilitation that are much more efficient on limited devices because the core representation is inherently compact.
Meng: Exactly. If we can run these models faster, it opens up ways to use action analysis in assistive tech without needing massive server farms.
Tom: It’s about efficiency translating directly into deployable technology, not just academic scores on a benchmark like NTU-sixty or NTU-one hundred twenty.
Jane: So, the big picture here is that we might get skeleton understanding systems that are powerful enough for many tasks but light enough to run everywhere.
Lu: And they’re still pushing the boundaries by showing how this compact token approach handles missing joints, though they do point out that unflagged zero-filled joints can still be a problem if you don't have a confidence signal.
Meng: That limitation about missing data is important for me. If the input is messy, we still need reliable signals upstream to handle it without the system breaking down at scale.
Tom: The title "Less is More" really captures that idea—that sometimes simplifying the structure of your model leads to a much better result for real-world application.
Jane: And this paper sets up an interesting challenge for future research: finding that sweet spot between extreme compactness and the need to capture all the necessary motion physics.
More episodes
- 2603.29765-Training-Free Dynamic Upcycling of Expert Language Models
- 2605.00778-Observable Performance Does Not Fully Reflect Adaptive System Organization: A Multi-Level Analysis of Gait Dynamics Under Occlusal Constraint
- 2605.06906-TraXion: Rethinking Pre-training Frameworks for Mobility and Beyond
- 2605.09165-Sparse Layers are Critical to Scaling Looped Language Models
- 2605.09439-Inverse Design for Conditional Distribution Matching
- 2605.10313-Signature Approach for Contextual Bandits with Nonlinear and Path-dependent Rewards: Regret Bound and Statistical Inference
- 2605.10878-Minimum Neural Weight Norm = Kolmogorov Complexity
- 2605.11617-MIST: Reliable Streaming Decision Trees for Online Class-Incremental Learning via McDiarmid Bound
- 2605.15085-From Data to Action: Accelerating Refinery Optimization with AI
- 2605.23500-GRTO: Bootstrapped Group Relative Tool Optimization for Referring Segmentation