Less is More: Compact-Token Masked Feature Prediction for Skeleton Representation Learning
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: We're looking at the paper: "Less is More".
Jane: Here's the gist of the paper. Less is More: Compact-Token Masked Feature Prediction for Skeleton Representation Learning proposes SLiM as a compact-token framework that unifies masked feature prediction and contrastive learning via a shared encoder The authors propose SLiM, a compact-token framework that unifies masked feature prediction and contrastive learning via a shared encoder By shifting the objective from raw coordinate reconstruction to decoder-free, teacher-guided feature prediction, SLiM breaks the reliance on dense tokenization and enables effective learning with a highly compact token grid Crucially, to prevent trivial shortcut learning arising from strong inter-joint dependencies of human, they introduce Semantic Tube Masking together with Skeleton-Aware Augmentations to enforce deep skeletal-temporal reasoning and anatomical consistency.
Tom: Extensive experiments across multiple downstream protocols demonstrate that SLiM achieves state-of-the-art performance while structurally reducing inference computation by 7.89× compared to dense-token MAE baselines.
Jane: Pruning redundant reasoning tokens Current skeleton representation learning paradigms face distinct limitations: Contrastive Learning (CL) often overlooks fine-grained motion details, while Masked Auto-Encoders (MAE) rely on coordinate-level reconstruction.
Tom: This reconstruction inherently demands dense token sequences and heavy decoders, wasting pre-training computation on discarded components and forcing downstream inference to process dense token grids.
Paper summary: Tom: So, looking at "Less is More: Compact-Token Masked Feature Prediction for Skeleton Representation Learning" and what we’ve seen, it boils down to using a decoder-free teacher-guided feature prediction objective combined with contrastive learning to learn skeleton representations.
Jane: The core idea seems to be that by targeting high level semantics instead of low level coordinates, you can break the dependency on dense tokenization, which is what really makes these current methods heavy.
Lu: It’s about using a compact token grid for the encoder and adding specific masking strategies like Semantic Tube Masking to keep that compact representation accurate enough for downstream tasks.
Meng: From an engineering standpoint, the seven point eight nine times reduction in inference cost is the big win because it shows a clear path to deploying these models on devices where computation is limited.
Tom: And they are showing that this compact token approach still achieves state of the art accuracy on several protocols while being structurally much lighter than dense token MAE baselines.
Jane: So, for someone listening who doesn't read the math, it means we can potentially get better action understanding systems that use less power and run faster without losing too much accuracy.
Conclusion: Tom: So, we're wrapping up this look at "Less is More: Compact-Token Masked Feature Prediction for Skeleton Representation Learning." Essentially, these authors are proposing a way to learn how skeletons move using much less data and less computation than what’s currently standard.
Jane: That sounds really simple on the surface. So, what is the main trick they used to make it work with fewer tokens?
Lu: The main trick is moving away from trying to reconstruct every single joint coordinate directly. Instead, they use this shared encoder—a compact token grid—to predict high-level features instead of low-level coordinates.
Meng: Because they aren't reconstructing dense points, it means the whole setup is much leaner computationally, which is what I care about when we’re trying to deploy models on real hardware.
Tom: Right, and they did this by combining two things: masked feature prediction and contrastive learning all through one shared encoder. That’s the framework they built.
Jane: But if it’s just a compact grid, won't that lose the fine motion details we need? What stops them from just getting blurry results?
Lu: They actually added specific techniques to stop that. They introduced Semantic Tube Masking and Skeleton-Aware Augmentations to make sure the model learns deep temporal reasoning and anatomical consistency instead of just memorizing a few pixels.
Meng: I’ve seen those kinds of augmentation tricks before, but if it helps keep the representation robust under real-world noise, that makes sense for production readiness.
Tom: The results they show are pretty strong too. They managed to beat some existing methods on several tasks while also cutting the inference cost by almost eight times compared to those dense token methods.
Jane: So, what does this mean practically? For someone who just listens to the show, what’s the real-world impact of learning skeletons with less data and less power?
Lu: It suggests that you can build action understanding systems for things like robotics or rehabilitation that are much more efficient on limited devices because the core representation is inherently compact.
Meng: Exactly. If we can run these models faster, it opens up ways to use action analysis in assistive tech without needing massive server farms.
Tom: It’s about efficiency translating directly into deployable technology, not just academic scores on a benchmark like NTU-sixty or NTU-one hundred twenty.
Jane: So, the big picture here is that we might get skeleton understanding systems that are powerful enough for many tasks but light enough to run everywhere.
Lu: And they’re still pushing the boundaries by showing how this compact token approach handles missing joints, though they do point out that unflagged zero-filled joints can still be a problem if you don't have a confidence signal.
Meng: That limitation about missing data is important for me. If the input is messy, we still need reliable signals upstream to handle it without the system breaking down at scale.
Tom: The title "Less is More" really captures that idea—that sometimes simplifying the structure of your model leads to a much better result for real-world application.
Jane: And this paper sets up an interesting challenge for future research: finding that sweet spot between extreme compactness and the need to capture all the necessary motion physics.
Information & Electronics Research Institute of KAIST
cs.CV
Submitted: 2026-03-11
Updated: 2026-11-05
Comments: NeurIPS 2026. Please visit our project page at https://kaist-viclab.github.io/SLiM_site/
Project page: https://kaist-viclab.github.io/SLiM_site/Abstract
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 100/100
The gist: Less is More: Compact-Token Masked Feature Prediction for Skeleton Representation Learning proposes SLiM as a compact-token framework that unifies masked feature prediction and contrastive learning…
Key concepts
- Compact Token Grid
- Instead of using dense sequences of tokens, SLiM employs a much smaller grid (e.g., 8x25 tokens) to represent skeletal data. This compact structure is key to drastically lowering the computational cost during the prediction phase compared to traditional methods.
- Masked Feature Prediction (MFP)
- This involves training a model to predict features of masked parts of a skeleton sequence, guided by a teacher network's output. This replaces raw coordinate reconstruction, allowing the model to learn high-level semantics instead of just precise joint positions.
- Semantic Tube Masking (STM)
- To prevent shortcuts from overly dependent joints, STM occludes connected anatomical regions across consecutive frames. This forces the model to perform deeper skeletal-temporal reasoning and ensures anatomical consistency during training.
Terminology
Summary
Less is More: Compact-Token Masked Feature Prediction for Skeleton Representation Learning proposes SLiM as a compact-token framework that unifies masked feature prediction and contrastive learning via a shared encoder The authors propose SLiM, a compact-token framework that unifies masked feature prediction and contrastive learning via a shared encoder By shifting the objective from raw coordinate reconstruction to decoder-free, teacher-guided feature prediction, SLiM breaks the reliance on dense tokenization and enables effective learning with a highly compact token grid Crucially, to prevent trivial shortcut learning arising from strong inter-joint dependencies of human, they introduce Semantic Tube Masking together with Skeleton-Aware Augmentations to enforce deep skeletal-temporal reasoning and anatomical consistency. p. 0 By shifting the objective from raw coordinate reconstruction to decoder-free, teacher-guided feature
prediction, SLiM breaks the reliance on dense tokenization and enables effective
learning with a highly compact token grid. Crucially, to prevent trivial shortcut
learning arising from strong inter-joint dependencies of human, we introduce Semantic Tube Masking together with Skeleton-Aware Augmentations to enforce
deep skeletal-temporal reasoning and anatomical consistency. Crucially, to prevent trivial shortcut
learning arising from strong inter-joint dependencies of human, we introduce Semantic Tube Masking together with Skeleton-Aware Augmentations to enforce
deep skeletal-temporal reasoning and anatomical consistency. Extensive experiments across multiple downstream protocols demonstrate that SLiM achieves
state-of-the-art performance while structurally reducing inference computation by
7. Extensive experiments across multiple downstream protocols demonstrate that SLiM achieves
state-of-the-art performance while structurally reducing inference computation by
7. 89× compared to dense-token MAE baselines.Where the paper says it
Where the paper says it
Where the paper says it
Pruning redundant reasoning tokens
Current skeleton representation learning paradigms face distinct limitations: Contrastive Learning (CL) often overlooks fine-grained motion details, while Masked Auto-Encoders (MAE) rely on coordinate-level reconstruction. This reconstruction inherently demands dense token sequences and heavy decoders, wasting pre-training computation on discarded components and forcing downstream inference to process dense token grids. p. 0 io/SLiM_site/
Abstract
Current skeleton representation learning paradigms face distinct limitations: Contrastive Learning (CL) often overlooks fine-grained motion details, while Masked
Auto-Encoders (MAE) rely on coordinate-level reconstruction. This reconstruction inherently demands dense token sequences and heavy decoders, wasting
pre-training computation on discarded components and forcing downstream inference to process dense token grids. To resolve these bottlenecks, we propose SLiM
(Skeleton Less is More), a compact-token framework that unifies masked feature
prediction and contrastive learning via a shared encoder. By shifting the objective from raw coordinate reconstruction to decoder-free, teacher-guided feature
prediction, SLiM breaks the reliance on dense tokenization and enables effective
learning with a highly compact token grid. By reformulating the objective
as a decoder-free, teacher-guided feature
prediction task (Fig. 2(b)), SLiM shifts the
target from low-level coordinates to highlevel semantics. 2(b)), SLiM shifts the
target from low-level coordinates to highlevel semantics. This paradigm shift completely eliminates the need for dense tokenization and heavy decoders.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
How the training signal is built
SLiM jointly optimizes masked feature prediction and global-local contrastive learning as Ltotal = LMFP + λLGLCL, where they empirically set λ = 1.0 for all experiments. p. 0 Crucially, to prevent trivial shortcut
learning arising from strong inter-joint dependencies of human, we introduce Semantic Tube Masking together with Skeleton-Aware Augmentations to enforce
deep skeletal-temporal reasoning and anatomical consistency. Extensive experiments across multiple downstream protocols demonstrate that SLiM achieves
state-of-the-art performance while structurally reducing inference computation by
7. Global-local contrastive learning (GLCL). We use the teacher network’s global class token from
XG1 as the anchor and align it with student network predictions from the other global and local views. The GLCL loss is:
LGLCL = −
1
|V|
X
Xk∈V
X
K
c=1
sgp
t
G1,cls(c)
log p
s
k,cls(c), (10)
where p
t
G1,cls and p
s
k,cls are the K-dimensional probability distributions obtained in the same
manner as Eq. This encourages temporal scale invariance by aligning local clips with the global
semantic context provided by the teacher network.Where the paper says it
Where the paper says it
Where the paper says it
Where the gains come from
The structural efficiency gain of SLiM is quantified by reducing inference computation by 7.89× compared to dense-token MAE-based methods. p. 7 SLiM reduces encoder inference cost
by 7. 89× compared to dense-token MAE-based methods. SLiM reduces encoder inference cost
by 7. 89× compared to dense-token MAE-based methods. Although the decoder is discarded after pre-training, downstream
inference still requires dense encoder tokens. (b) SLiM replaces coordinate reconstruction with
decoder-free masked feature prediction under a compact token grid (8 × J), using stochastic masktoken corruption and contrastive learning during pre-training. However, compact feature prediction alone is vulnerable to shortcut learning; strong skeletal-temporal
correlations allow independent joint masking [1, 22, 31] to be trivially solved via local interpolation. To enforce true action-level reasoning, we introduce Semantic Tube Masking, which occludes connected anatomical regions over consecutive frames. To enforce true action-level reasoning, we introduce Semantic Tube Masking, which occludes connected anatomical regions over consecutive frames. For the contrastive branch, we propose SkeletonAware Augmentations such as skeleton-aware rotation, skeleton-aware mirroring, and bone-aware
scaling to generate diverse views while strictly preserving anatomical validity.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Token budget and computational cost comparison
Table 3 provides a token budget and computational cost comparison on NTU-60, showing that SLiM reduces encoder inference cost by 7.89× compared to dense-token MAE-based methods. Specifically, SLiM uses an 8 × 25 encoder token grid and requires only 3. p. 7 1, coordinatereconstruction MAE baselines degrade noticeably when the temporal token budget is reduced,
whereas SLiM maintains strong accuracy under compact token budgets. Table 3 further provides the
computational breakdown: SLiM uses an 8 × 25 encoder token grid and requires only 3. Table 3 further provides the
computational breakdown: SLiM uses an 8 × 25 encoder token grid and requires only 3. 59 GFLOPs
for downstream inference, yielding a 7. 59 GFLOPs
for downstream inference, yielding a 7. 89× reduction compared to dense-token MAE-based methods
with 30 × 25 tokens. Since MFP predicts teacher features rather than raw coordinates,
SLiM removes the reconstruction decoder during pre-training. The downstream inference cost is then
governed by the compact encoder token grid used for evaluation.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Performance comparison
Table 1 reports Top-1 accuracy under the standard linear evaluation protocol, where including concurrent AMR, SLiM obtains the best result on four of five protocols, exceeding AMR by 0.5, 0.9, 0.1, and 1.7%-points on NTU-60 X-Sub, NTU-60 X-View, NTU-120 X-Sub, and NTU-120 X-Set. p. 7 Including concurrent
AMR [29], SLiM obtains the best result on four of five protocols, exceeding AMR by 0. 7%-points on NTU-60 X-Sub, NTU-60 X-View, NTU-120 X-Sub, and NTU-120 X-Set,
respectively. Table 5(a) evaluates STM’s dual role in generating masked global
views (MFP) and perturbing local views (GLCL). Replacing STM with conventional joint masking
[1, 22, 31] in either role degrades X-Sub accuracy by up to 3. Crucially, to prevent trivial shortcut
learning arising from strong inter-joint dependencies of human, we introduce Semantic Tube Masking together with Skeleton-Aware Augmentations to enforce
deep skeletal-temporal reasoning and anatomical consistency. Extensive experiments across multiple downstream protocols demonstrate that SLiM achieves
state-of-the-art performance while structurally reducing inference computation by
7. Bold and underline indicate the best and second-best results, respectively. Method Publication NTU-60 NTU-120 PKU-MMD II
X-Sub X-View X-Sub X-Set X-Sub
Other skeleton SSL baselines:
GL-Transformer [13] ECCV’22 76.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Robustness analysis
In the robustness analysis, SLiM provides a mechanism for declared missingness by routing unavailable joints through the learned mask token. However, unflagged parent- or zero-filled joints can cause severe degradation, and this conversion requires a reliable upstream confidence or visibility signal. The results show that degraded pre-training substantially improves the matched unflagged zero-fill condition when missingness can be declared through confidence or visibility information. p. 9 D Skeleton-Aware Augmentations (SAA):
Rotation (Algorithm 2), mirroring (Algorithm 3), and scaling (Algorithm 4)
Sec. E
Additional Results and Discussion:
Token-Budget Study (Table 7), detailed cost comparison (Table 8),
transfer learning (Table 9), and controlled prediction-target analysis (Table 10),
missing/noisy-joint robustness and degraded pre-training (Tables 11–12)
Sec.Where the paper says it
Limitations and broader impact
SLiM targets efficient skeleton representation learning under a compact skeletal-temporal token budget, but actions requiring subtle high-frequency motion may benefit from denser temporal tokens. The empirical scope is limited to 3D joint sequences from three benchmarks with a fixed 25-joint topology. p. 6 1 Datasets and Protocols
We evaluate SLiM on three standard skeleton-based action recognition benchmarks: NTU RGB+D 60
(NTU-60) [26], NTU RGB+D 120 (NTU-120) [19], and PKU-MMD II [18]. NTU-60 contains 56,880
skeleton sequences from 60 action classes and provides Cross-Subject (X-Sub) and Cross-View (XView) protocols. 89× compared to dense-token MAE baselines. 1 Introduction
Skeleton data provides a compact and informative representation for human action understanding,
offering inherent robustness to background and viewpoint changes compared to RGB videos [14,
32, 40]. 1 Introduction
Skeleton data provides a compact and informative representation for human action understanding,
offering inherent robustness to background and viewpoint changes compared to RGB videos [14,
32, 40]. While supervised skeleton-based action recognition has achieved strong performance
[4, 5, 8], it depends on costly annotations and often generalizes poorly to unseen domains.Where the paper says it
Where the paper says it
Where the paper says it
Compute resources and existing assets
SLiM is pre-trained on 4 NVIDIA RTX A6000 GPUs with a total batch size of 768, while the final model is pre-trained for 150 epochs. The datasets used are NTU RGB+D 60 [26], NTU RGB+D 120 [19], and PKU-MMD II [18] as public benchmark datasets. p. 6 4 Experimental Results
4. 1 Datasets and Protocols
We evaluate SLiM on three standard skeleton-based action recognition benchmarks: NTU RGB+D 60
(NTU-60) [26], NTU RGB+D 120 (NTU-120) [19], and PKU-MMD II [18]. Building upon these promising
results, future work will explore extending our compact-token paradigm to more challenging scenarios,
such as noisy 2D poses, missing joints, and heterogeneous skeleton topologies. Acknowledgments
This work was supported by the National Research Foundation of Korea(NRF) grant funded by the
Korea government(MSIT) (RS-2026-25484549).Where the paper says it
Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Compact Token Framework for Skeleton Representation Learning: The SLiM framework unifies masked feature prediction and contrastive learning via a shared encoder to break reliance on dense tokenization, enabling effective learning with a highly compact token grid. This structural change reduces inference computation by 7.89 times compared to dense-token MAE baselines.
-
Decoder-Free Masked Feature Prediction: SLiM reformulates the objective as a decoder-free, teacher-guided feature prediction task, shifting the target from low-level coordinates to highlevel semantics, which completely eliminates the need for dense tokenization and heavy decoders.
-
Semantic Tube Masking for Shortcut Prevention: By introducing Semantic Tube Masking, the system occludes connected anatomical regions over consecutive frames to prevent trivial shortcut learning arising from strong inter-joint dependencies of human, ensuring deep skeletal-temporal reasoning and anatomical consistency.
-
Skeleton-Aware Augmentations for Anatomical Validity: The system utilizes Skeleton-Aware Augmentations such as skeleton-aware rotation, mirroring, and bone-aware scaling to generate diverse views while strictly preserving anatomical validity through geometric transformations that maintain the articulated structure of the human body.
-
Enhanced Representation Robustness: The system demonstrates robustness to declared missingness by routing unavailable joints through a learned mask token, and its performance is improved when pre-training data includes unflagged zero-fill or Gaussian noise, providing a mechanism for handling certain forms of corruption.
Abstract
Current skeleton representation learning paradigms face distinct limitations: Contrastive Learning (CL) often overlooks fine-grained motion details, while Masked Auto-Encoders (MAE) rely on coordinate-level reconstruction. This reconstruction inherently demands dense token sequences and heavy decoders, wasting pre-training computation on discarded components and forcing downstream inference to process dense token grids. To resolve these bottlenecks, we propose SLiM (Skeleton Less is More), a compact-token framework that unifies masked feature prediction and contrastive learning via a shared encoder. By shifting the objective from raw coordinate reconstruction to decoder-free, teacher-guided feature prediction, SLiM breaks the reliance on dense tokenization and enables effective learning with a highly compact token grid. Crucially, to prevent trivial shortcut learning arising from strong inter-joint dependencies of human, we introduce Semantic Tube Masking together with Skeleton-Aware Augmentations to enforce deep skeletal-temporal reasoning and anatomical consistency. Extensive experiments across multiple downstream protocols demonstrate that SLiM achieves state-of-the-art performance while structurally reducing inference computation by 7.89 times compared to dense-token MAE baselines.
Sources
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- HYperbolic Self-Paced Learning for Self-Supervised Skeleton-based Action Representations
- Part Aware Contrastive Learning for Self-Supervised Action Recognition
- PKU-MMD: A Large Scale Benchmark for Continuous Multi-Modal Human Action Understanding
- SGDR: Stochastic Gradient Descent with Warm Restarts
- Decoupled Weight Decay Regularization
- DINOv2: Learning Robust Visual Features without Supervision
- iBOT: Image BERT Pre-Training with Online Tokenizer
Related papers
- TEA: Text Encoder Alignment for Robust Concept Erasure in Text-to-Image Models
- Do Egocentric Video-Language Models Capture Both Hand- and Object-Centric Cues?
- Aura: Consistent Multi-Subject Video Generation via VLM-Grounded Semantic Alignment
- PVRF: All-in-one Adverse Weather Removal via Prior-modulated and Velocity-constrained Rectified Flow
- DLWM: Diverse Latent World Models for Efficient Multimodal Reasoning
- Out of Sight, Still in Mind: Token Compression for Omni-LLMs