Less is More: Compact-Token Masked Feature Prediction for Skeleton Representation Learning

summary

The gist

Less is More: Compact-Token Masked Feature Prediction for Skeleton Representation Learning proposes SLiM as a compact-token framework that unifies masked feature prediction and contrastive learning…

In short

SLiM proposes a compact-token framework to unify masked feature prediction and contrastive learning for skeleton representation. It shifts the focus from dense coordinate reconstruction to teacher-guided semantic feature prediction using a small token grid, significantly reducing inference computation while maintaining state-of-the-art performance.

Key concepts

Compact Token Grid
Instead of using dense sequences of tokens, SLiM employs a much smaller grid (e.g., 8x25 tokens) to represent skeletal data. This compact structure is key to drastically lowering the computational cost during the prediction phase compared to traditional methods.
Masked Feature Prediction (MFP)
This involves training a model to predict features of masked parts of a skeleton sequence, guided by a teacher network's output. This replaces raw coordinate reconstruction, allowing the model to learn high-level semantics instead of just precise joint positions.
Semantic Tube Masking (STM)
To prevent shortcuts from overly dependent joints, STM occludes connected anatomical regions across consecutive frames. This forces the model to perform deeper skeletal-temporal reasoning and ensures anatomical consistency during training.

Terminology used across episodes

This episode discusses

The paper

Less is More: Compact-Token Masked Feature Prediction for Skeleton Representation Learning · Read on arXiv

Information & Electronics Research Institute of KAIST

Current skeleton representation learning paradigms face distinct limitations: Contrastive Learning (CL) often overlooks fine-grained motion details, while Masked Auto-Encoders (MAE) rely on coordinate-level reconstruction. This reconstruction inherently demands dense token sequences and heavy decoders, wasting pre-training computation on discarded components and forcing downstream inference to process dense token grids. To resolve these bottlenecks, we propose SLiM (Skeleton Less is More), a compact-token framework that unifies masked feature prediction and contrastive learning via a shared encoder. By shifting the objective from raw coordinate reconstruction to decoder-free, teacher-guided feature prediction, SLiM breaks the reliance on dense tokenization and enables effective learning with a highly compact token grid. Crucially, to prevent trivial shortcut learning arising from strong inter-joint dependencies of human, we introduce Semantic Tube Masking together with Skeleton-Aware Augmentations to enforce deep skeletal-temporal reasoning and anatomical consistency. Extensive experiments across multiple downstream protocols demonstrate that SLiM achieves state-of-the-art performance while structurally reducing inference computation by 7.89 times compared to dense-token MAE baselines.

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "Less is More".

Jane: Here's the gist of the paper. Less is More: Compact-Token Masked Feature Prediction for Skeleton Representation Learning proposes SLiM as a compact-token framework that unifies masked feature prediction and contrastive learning via a shared encoder The authors propose SLiM, a compact-token framework that unifies masked feature prediction and contrastive learning via a shared encoder By shifting the objective from raw coordinate reconstruction to decoder-free, teacher-guided feature prediction, SLiM breaks the reliance on dense tokenization and enables effective learning with a highly compact token grid Crucially, to prevent trivial shortcut learning arising from strong inter-joint dependencies of human, they introduce Semantic Tube Masking together with Skeleton-Aware Augmentations to enforce deep skeletal-temporal reasoning and anatomical consistency.

Tom: Extensive experiments across multiple downstream protocols demonstrate that SLiM achieves state-of-the-art performance while structurally reducing inference computation by 7.89× compared to dense-token MAE baselines.

Jane: Pruning redundant reasoning tokens Current skeleton representation learning paradigms face distinct limitations: Contrastive Learning (CL) often overlooks fine-grained motion details, while Masked Auto-Encoders (MAE) rely on coordinate-level reconstruction.

Tom: This reconstruction inherently demands dense token sequences and heavy decoders, wasting pre-training computation on discarded components and forcing downstream inference to process dense token grids.

Paper summary: Tom: So, looking at "Less is More: Compact-Token Masked Feature Prediction for Skeleton Representation Learning" and what we’ve seen, it boils down to using a decoder-free teacher-guided feature prediction objective combined with contrastive learning to learn skeleton representations.

Jane: The core idea seems to be that by targeting high level semantics instead of low level coordinates, you can break the dependency on dense tokenization, which is what really makes these current methods heavy.

Lu: It’s about using a compact token grid for the encoder and adding specific masking strategies like Semantic Tube Masking to keep that compact representation accurate enough for downstream tasks.

Meng: From an engineering standpoint, the seven point eight nine times reduction in inference cost is the big win because it shows a clear path to deploying these models on devices where computation is limited.

Tom: And they are showing that this compact token approach still achieves state of the art accuracy on several protocols while being structurally much lighter than dense token MAE baselines.

Jane: So, for someone listening who doesn't read the math, it means we can potentially get better action understanding systems that use less power and run faster without losing too much accuracy.

Conclusion: Tom: So, we're wrapping up this look at "Less is More: Compact-Token Masked Feature Prediction for Skeleton Representation Learning." Essentially, these authors are proposing a way to learn how skeletons move using much less data and less computation than what’s currently standard.

Jane: That sounds really simple on the surface. So, what is the main trick they used to make it work with fewer tokens?

Lu: The main trick is moving away from trying to reconstruct every single joint coordinate directly. Instead, they use this shared encoder—a compact token grid—to predict high-level features instead of low-level coordinates.

Meng: Because they aren't reconstructing dense points, it means the whole setup is much leaner computationally, which is what I care about when we’re trying to deploy models on real hardware.

Tom: Right, and they did this by combining two things: masked feature prediction and contrastive learning all through one shared encoder. That’s the framework they built.

Jane: But if it’s just a compact grid, won't that lose the fine motion details we need? What stops them from just getting blurry results?

Lu: They actually added specific techniques to stop that. They introduced Semantic Tube Masking and Skeleton-Aware Augmentations to make sure the model learns deep temporal reasoning and anatomical consistency instead of just memorizing a few pixels.

Meng: I’ve seen those kinds of augmentation tricks before, but if it helps keep the representation robust under real-world noise, that makes sense for production readiness.

Tom: The results they show are pretty strong too. They managed to beat some existing methods on several tasks while also cutting the inference cost by almost eight times compared to those dense token methods.

Jane: So, what does this mean practically? For someone who just listens to the show, what’s the real-world impact of learning skeletons with less data and less power?

Lu: It suggests that you can build action understanding systems for things like robotics or rehabilitation that are much more efficient on limited devices because the core representation is inherently compact.

Meng: Exactly. If we can run these models faster, it opens up ways to use action analysis in assistive tech without needing massive server farms.

Tom: It’s about efficiency translating directly into deployable technology, not just academic scores on a benchmark like NTU-sixty or NTU-one hundred twenty.

Jane: So, the big picture here is that we might get skeleton understanding systems that are powerful enough for many tasks but light enough to run everywhere.

Lu: And they’re still pushing the boundaries by showing how this compact token approach handles missing joints, though they do point out that unflagged zero-filled joints can still be a problem if you don't have a confidence signal.

Meng: That limitation about missing data is important for me. If the input is messy, we still need reliable signals upstream to handle it without the system breaking down at scale.

Tom: The title "Less is More" really captures that idea—that sometimes simplifying the structure of your model leads to a much better result for real-world application.

Jane: And this paper sets up an interesting challenge for future research: finding that sweet spot between extreme compactness and the need to capture all the necessary motion physics.

More episodes

← Home