Less is More: Compact-Token Masked Feature Prediction for Skeleton Representation Learning

arXiv:2603.10648 · cs.CV · Submitted 2026-03-11 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "Less is More".

Jane: Here's the gist of the paper. Less is More: Compact-Token Masked Feature Prediction for Skeleton Representation Learning proposes SLiM as a compact-token framework that unifies masked feature prediction and contrastive learning via a shared encoder The authors propose SLiM, a compact-token framework that unifies masked feature prediction and contrastive learning via a shared encoder By shifting the objective from raw coordinate reconstruction to decoder-free, teacher-guided feature prediction, SLiM breaks the reliance on dense tokenization and enables effective learning with a highly compact token grid Crucially, to prevent trivial shortcut learning arising from strong inter-joint dependencies of human, they introduce Semantic Tube Masking together with Skeleton-Aware Augmentations to enforce deep skeletal-temporal reasoning and anatomical consistency.

Tom: Extensive experiments across multiple downstream protocols demonstrate that SLiM achieves state-of-the-art performance while structurally reducing inference computation by 7.89× compared to dense-token MAE baselines.

Jane: Pruning redundant reasoning tokens Current skeleton representation learning paradigms face distinct limitations: Contrastive Learning (CL) often overlooks fine-grained motion details, while Masked Auto-Encoders (MAE) rely on coordinate-level reconstruction.

Tom: This reconstruction inherently demands dense token sequences and heavy decoders, wasting pre-training computation on discarded components and forcing downstream inference to process dense token grids.

Paper summary: Tom: So, looking at "Less is More: Compact-Token Masked Feature Prediction for Skeleton Representation Learning" and what we’ve seen, it boils down to using a decoder-free teacher-guided feature prediction objective combined with contrastive learning to learn skeleton representations.

Jane: The core idea seems to be that by targeting high level semantics instead of low level coordinates, you can break the dependency on dense tokenization, which is what really makes these current methods heavy.

Lu: It’s about using a compact token grid for the encoder and adding specific masking strategies like Semantic Tube Masking to keep that compact representation accurate enough for downstream tasks.

Meng: From an engineering standpoint, the seven point eight nine times reduction in inference cost is the big win because it shows a clear path to deploying these models on devices where computation is limited.

Tom: And they are showing that this compact token approach still achieves state of the art accuracy on several protocols while being structurally much lighter than dense token MAE baselines.

Jane: So, for someone listening who doesn't read the math, it means we can potentially get better action understanding systems that use less power and run faster without losing too much accuracy.

Conclusion: Tom: So, we're wrapping up this look at "Less is More: Compact-Token Masked Feature Prediction for Skeleton Representation Learning." Essentially, these authors are proposing a way to learn how skeletons move using much less data and less computation than what’s currently standard.

Jane: That sounds really simple on the surface. So, what is the main trick they used to make it work with fewer tokens?

Lu: The main trick is moving away from trying to reconstruct every single joint coordinate directly. Instead, they use this shared encoder—a compact token grid—to predict high-level features instead of low-level coordinates.

Meng: Because they aren't reconstructing dense points, it means the whole setup is much leaner computationally, which is what I care about when we’re trying to deploy models on real hardware.

Tom: Right, and they did this by combining two things: masked feature prediction and contrastive learning all through one shared encoder. That’s the framework they built.

Jane: But if it’s just a compact grid, won't that lose the fine motion details we need? What stops them from just getting blurry results?

Lu: They actually added specific techniques to stop that. They introduced Semantic Tube Masking and Skeleton-Aware Augmentations to make sure the model learns deep temporal reasoning and anatomical consistency instead of just memorizing a few pixels.

Meng: I’ve seen those kinds of augmentation tricks before, but if it helps keep the representation robust under real-world noise, that makes sense for production readiness.

Tom: The results they show are pretty strong too. They managed to beat some existing methods on several tasks while also cutting the inference cost by almost eight times compared to those dense token methods.

Jane: So, what does this mean practically? For someone who just listens to the show, what’s the real-world impact of learning skeletons with less data and less power?

Lu: It suggests that you can build action understanding systems for things like robotics or rehabilitation that are much more efficient on limited devices because the core representation is inherently compact.

Meng: Exactly. If we can run these models faster, it opens up ways to use action analysis in assistive tech without needing massive server farms.

Tom: It’s about efficiency translating directly into deployable technology, not just academic scores on a benchmark like NTU-sixty or NTU-one hundred twenty.

Jane: So, the big picture here is that we might get skeleton understanding systems that are powerful enough for many tasks but light enough to run everywhere.

Lu: And they’re still pushing the boundaries by showing how this compact token approach handles missing joints, though they do point out that unflagged zero-filled joints can still be a problem if you don't have a confidence signal.

Meng: That limitation about missing data is important for me. If the input is messy, we still need reliable signals upstream to handle it without the system breaking down at scale.

Tom: The title "Less is More" really captures that idea—that sometimes simplifying the structure of your model leads to a much better result for real-world application.

Jane: And this paper sets up an interesting challenge for future research: finding that sweet spot between extreme compactness and the need to capture all the necessary motion physics.

Information & Electronics Research Institute of KAIST

cs.CV

Submitted: 2026-03-11

Updated: 2026-11-05

Comments: NeurIPS 2026. Please visit our project page at https://kaist-viclab.github.io/SLiM_site/

Project page: https://kaist-viclab.github.io/SLiM_site/Abstract

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 100/100

The gist: Less is More: Compact-Token Masked Feature Prediction for Skeleton Representation Learning proposes SLiM as a compact-token framework that unifies masked feature prediction and contrastive learning…

Key concepts

Compact Token Grid
Instead of using dense sequences of tokens, SLiM employs a much smaller grid (e.g., 8x25 tokens) to represent skeletal data. This compact structure is key to drastically lowering the computational cost during the prediction phase compared to traditional methods.
Masked Feature Prediction (MFP)
This involves training a model to predict features of masked parts of a skeleton sequence, guided by a teacher network's output. This replaces raw coordinate reconstruction, allowing the model to learn high-level semantics instead of just precise joint positions.
Semantic Tube Masking (STM)
To prevent shortcuts from overly dependent joints, STM occludes connected anatomical regions across consecutive frames. This forces the model to perform deeper skeletal-temporal reasoning and ensures anatomical consistency during training.

Terminology

Summary

Less is More: Compact-Token Masked Feature Prediction for Skeleton Representation Learning proposes SLiM as a compact-token framework that unifies masked feature prediction and contrastive learning via a shared encoder The authors propose SLiM, a compact-token framework that unifies masked feature prediction and contrastive learning via a shared encoder By shifting the objective from raw coordinate reconstruction to decoder-free, teacher-guided feature prediction, SLiM breaks the reliance on dense tokenization and enables effective learning with a highly compact token grid Crucially, to prevent trivial shortcut learning arising from strong inter-joint dependencies of human, they introduce Semantic Tube Masking together with Skeleton-Aware Augmentations to enforce deep skeletal-temporal reasoning and anatomical consistency. p. 0

Where the paper says it

By shifting the objective from raw coordinate reconstruction to decoder-free, teacher-guided feature prediction, SLiM breaks the reliance on dense tokenization and enables effective learning with a highly compact token grid. Crucially, to prevent trivial shortcut learning arising from strong inter-joint dependencies of human, we introduce Semantic Tube Masking together with Skeleton-Aware Augmentations to enforce deep skeletal-temporal reasoning and anatomical consistency.

Extensive experiments across multiple downstream protocols demonstrate that SLiM achieves state-of-the-art performance while structurally reducing inference computation by 7. p. 0
Where the paper says it

Crucially, to prevent trivial shortcut learning arising from strong inter-joint dependencies of human, we introduce Semantic Tube Masking together with Skeleton-Aware Augmentations to enforce deep skeletal-temporal reasoning and anatomical consistency. Extensive experiments across multiple downstream protocols demonstrate that SLiM achieves state-of-the-art performance while structurally reducing inference computation by 7.

89× compared to dense-token MAE baselines. p. 0
Where the paper says it

Extensive experiments across multiple downstream protocols demonstrate that SLiM achieves state-of-the-art performance while structurally reducing inference computation by 7. 89× compared to dense-token MAE baselines.

Pruning redundant reasoning tokens

Current skeleton representation learning paradigms face distinct limitations: Contrastive Learning (CL) often overlooks fine-grained motion details, while Masked Auto-Encoders (MAE) rely on coordinate-level reconstruction. This reconstruction inherently demands dense token sequences and heavy decoders, wasting pre-training computation on discarded components and forcing downstream inference to process dense token grids. p. 0

Where the paper says it

io/SLiM_site/ Abstract Current skeleton representation learning paradigms face distinct limitations: Contrastive Learning (CL) often overlooks fine-grained motion details, while Masked Auto-Encoders (MAE) rely on coordinate-level reconstruction. This reconstruction inherently demands dense token sequences and heavy decoders, wasting pre-training computation on discarded components and forcing downstream inference to process dense token grids.

To resolve these bottlenecks, SLiM (Skeleton Less is More), a compact-token framework that unifies masked feature prediction and contrastive learning via a shared encoder By reformulating the objective as a decoder-free, teacher-guided feature prediction task (Fig. p. 0
Where the paper says it

To resolve these bottlenecks, we propose SLiM (Skeleton Less is More), a compact-token framework that unifies masked feature prediction and contrastive learning via a shared encoder. By shifting the objective from raw coordinate reconstruction to decoder-free, teacher-guided feature prediction, SLiM breaks the reliance on dense tokenization and enables effective learning with a highly compact token grid.

2(b)), SLiM shifts the target from low-level coordinates to highlevel semantics. p. 1
Where the paper says it

By reformulating the objective as a decoder-free, teacher-guided feature prediction task (Fig. 2(b)), SLiM shifts the target from low-level coordinates to highlevel semantics.

This paradigm shift completely eliminates the need for dense tokenization and heavy decoders. p. 1
Where the paper says it

2(b)), SLiM shifts the target from low-level coordinates to highlevel semantics. This paradigm shift completely eliminates the need for dense tokenization and heavy decoders.

How the training signal is built

SLiM jointly optimizes masked feature prediction and global-local contrastive learning as Ltotal = LMFP + λLGLCL, where they empirically set λ = 1.0 for all experiments. p. 0

Where the paper says it

Crucially, to prevent trivial shortcut learning arising from strong inter-joint dependencies of human, we introduce Semantic Tube Masking together with Skeleton-Aware Augmentations to enforce deep skeletal-temporal reasoning and anatomical consistency. Extensive experiments across multiple downstream protocols demonstrate that SLiM achieves state-of-the-art performance while structurally reducing inference computation by 7.

The Masked Feature Prediction loss (LMFP) is formulated by applying the teacher network's output to the masked patch tokens, using a binary mask M in Eq. 5. The Global-Local Contrastive Learning loss (LGLCL) aligns the teacher network’s global class token from XG1 with student network predictions from other global and local views. p. 4
Where the paper says it

Global-local contrastive learning (GLCL). We use the teacher network’s global class token from XG1 as the anchor and align it with student network predictions from the other global and local views.

This encourages temporal scale invariance by aligning local clips with the global semantic context provided by the teacher network. p. 4
Where the paper says it

The GLCL loss is: LGLCL = − 1 |V| X Xk∈V X K c=1 sgp t G1,cls(c) log p s k,cls(c), (10) where p t G1,cls and p s k,cls are the K-dimensional probability distributions obtained in the same manner as Eq. This encourages temporal scale invariance by aligning local clips with the global semantic context provided by the teacher network.

Where the gains come from

The structural efficiency gain of SLiM is quantified by reducing inference computation by 7.89× compared to dense-token MAE-based methods. p. 7

Where the paper says it

SLiM reduces encoder inference cost by 7. 89× compared to dense-token MAE-based methods.

p. 7
Where the paper says it

SLiM reduces encoder inference cost by 7. 89× compared to dense-token MAE-based methods.

This reduction stems from using a compact encoder token grid, such as 8 × 25 tokens, instead of the dense token grids used in baselines. p. 2
Where the paper says it

Although the decoder is discarded after pre-training, downstream inference still requires dense encoder tokens. (b) SLiM replaces coordinate reconstruction with decoder-free masked feature prediction under a compact token grid (8 × J), using stochastic masktoken corruption and contrastive learning during pre-training.

Furthermore, Semantic Tube Masking (STM) is introduced to avoid interpolation-based shortcuts by occluding connected anatomical regions over consecutive frames. p. 1
Where the paper says it

However, compact feature prediction alone is vulnerable to shortcut learning; strong skeletal-temporal correlations allow independent joint masking [1, 22, 31] to be trivially solved via local interpolation. To enforce true action-level reasoning, we introduce Semantic Tube Masking, which occludes connected anatomical regions over consecutive frames.

Skeleton-Aware Augmentations (SAA) are used to generate diverse views while strictly preserving anatomical validity. p. 1
Where the paper says it

To enforce true action-level reasoning, we introduce Semantic Tube Masking, which occludes connected anatomical regions over consecutive frames. For the contrastive branch, we propose SkeletonAware Augmentations such as skeleton-aware rotation, skeleton-aware mirroring, and bone-aware scaling to generate diverse views while strictly preserving anatomical validity.

Token budget and computational cost comparison

Table 3 provides a token budget and computational cost comparison on NTU-60, showing that SLiM reduces encoder inference cost by 7.89× compared to dense-token MAE-based methods. Specifically, SLiM uses an 8 × 25 encoder token grid and requires only 3. p. 7

Where the paper says it

1, coordinatereconstruction MAE baselines degrade noticeably when the temporal token budget is reduced, whereas SLiM maintains strong accuracy under compact token budgets. Table 3 further provides the computational breakdown: SLiM uses an 8 × 25 encoder token grid and requires only 3.

59 GFLOPs for downstream inference, yielding a 7. p. 7
Where the paper says it

Table 3 further provides the computational breakdown: SLiM uses an 8 × 25 encoder token grid and requires only 3. 59 GFLOPs for downstream inference, yielding a 7.

89× reduction compared to dense-token MAE-based methods with 30 × 25 tokens. p. 7
Where the paper says it

59 GFLOPs for downstream inference, yielding a 7. 89× reduction compared to dense-token MAE-based methods with 30 × 25 tokens.

Table 8 provides a module-level cost breakdown where SLiM removes the coordinate-reconstruction decoder, and its target/auxiliary entry is the EMA teacher forward on compact tokens. p. 4
Where the paper says it

Since MFP predicts teacher features rather than raw coordinates, SLiM removes the reconstruction decoder during pre-training. The downstream inference cost is then governed by the compact encoder token grid used for evaluation.

Performance comparison

Table 1 reports Top-1 accuracy under the standard linear evaluation protocol, where including concurrent AMR, SLiM obtains the best result on four of five protocols, exceeding AMR by 0.5, 0.9, 0.1, and 1.7%-points on NTU-60 X-Sub, NTU-60 X-View, NTU-120 X-Sub, and NTU-120 X-Set. p. 7

Where the paper says it

Including concurrent AMR [29], SLiM obtains the best result on four of five protocols, exceeding AMR by 0. 7%-points on NTU-60 X-Sub, NTU-60 X-View, NTU-120 X-Sub, and NTU-120 X-Set, respectively.

Table 5(a) evaluates the dual role of Semantic Tube Masking (STM), confirming that replacing STM with conventional joint masking degrades X-Sub accuracy by up to 3. p. 8
Where the paper says it

Table 5(a) evaluates STM’s dual role in generating masked global views (MFP) and perturbing local views (GLCL). Replacing STM with conventional joint masking [1, 22, 31] in either role degrades X-Sub accuracy by up to 3.

5%-point. Table 9 reports transfer learning performance on PKU-MMD II, where SLiM achieves 72. p. 0
Where the paper says it

Crucially, to prevent trivial shortcut learning arising from strong inter-joint dependencies of human, we introduce Semantic Tube Masking together with Skeleton-Aware Augmentations to enforce deep skeletal-temporal reasoning and anatomical consistency. Extensive experiments across multiple downstream protocols demonstrate that SLiM achieves state-of-the-art performance while structurally reducing inference computation by 7.

7% and 75.3% accuracy when pre-trained on NTU-60 and NTU-120, respectively. p. 6
Where the paper says it

Bold and underline indicate the best and second-best results, respectively. Method Publication NTU-60 NTU-120 PKU-MMD II X-Sub X-View X-Sub X-Set X-Sub Other skeleton SSL baselines: GL-Transformer [13] ECCV’22 76.

Robustness analysis

In the robustness analysis, SLiM provides a mechanism for declared missingness by routing unavailable joints through the learned mask token. However, unflagged parent- or zero-filled joints can cause severe degradation, and this conversion requires a reliable upstream confidence or visibility signal. The results show that degraded pre-training substantially improves the matched unflagged zero-fill condition when missingness can be declared through confidence or visibility information. p. 9

Where the paper says it

D Skeleton-Aware Augmentations (SAA): Rotation (Algorithm 2), mirroring (Algorithm 3), and scaling (Algorithm 4) Sec. E Additional Results and Discussion: Token-Budget Study (Table 7), detailed cost comparison (Table 8), transfer learning (Table 9), and controlled prediction-target analysis (Table 10), missing/noisy-joint robustness and degraded pre-training (Tables 11–12) Sec.

Limitations and broader impact

SLiM targets efficient skeleton representation learning under a compact skeletal-temporal token budget, but actions requiring subtle high-frequency motion may benefit from denser temporal tokens. The empirical scope is limited to 3D joint sequences from three benchmarks with a fixed 25-joint topology. p. 6

Where the paper says it

1 Datasets and Protocols We evaluate SLiM on three standard skeleton-based action recognition benchmarks: NTU RGB+D 60 (NTU-60) [26], NTU RGB+D 120 (NTU-120) [19], and PKU-MMD II [18]. NTU-60 contains 56,880 skeleton sequences from 60 action classes and provides Cross-Subject (X-Sub) and Cross-View (XView) protocols.

Efficient skeleton representation learning may benefit applications such as human-computer interaction, robotics, rehabilitation support, and assistive systems by reducing the computational cost of action understanding. p. 0
Where the paper says it

89× compared to dense-token MAE baselines. 1 Introduction Skeleton data provides a compact and informative representation for human action understanding, offering inherent robustness to background and viewpoint changes compared to RGB videos [14, 32, 40].

The paper notes that wall-clock behavior depends on hardware and implementation. p. 0
Where the paper says it

1 Introduction Skeleton data provides a compact and informative representation for human action understanding, offering inherent robustness to background and viewpoint changes compared to RGB videos [14, 32, 40]. While supervised skeleton-based action recognition has achieved strong performance [4, 5, 8], it depends on costly annotations and often generalizes poorly to unseen domains.

Compute resources and existing assets

SLiM is pre-trained on 4 NVIDIA RTX A6000 GPUs with a total batch size of 768, while the final model is pre-trained for 150 epochs. The datasets used are NTU RGB+D 60 [26], NTU RGB+D 120 [19], and PKU-MMD II [18] as public benchmark datasets. p. 6

Where the paper says it

4 Experimental Results 4. 1 Datasets and Protocols We evaluate SLiM on three standard skeleton-based action recognition benchmarks: NTU RGB+D 60 (NTU-60) [26], NTU RGB+D 120 (NTU-120) [19], and PKU-MMD II [18].

The work was supported by the National Research Foundation of Korea(NRF) grant funded by the Korea government(MSIT) (RS-2026-25484549). p. 8
Where the paper says it

Building upon these promising results, future work will explore extending our compact-token paradigm to more challenging scenarios, such as noisy 2D poses, missing joints, and heterogeneous skeleton topologies. Acknowledgments This work was supported by the National Research Foundation of Korea(NRF) grant funded by the Korea government(MSIT) (RS-2026-25484549).

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Compact Token Framework for Skeleton Representation Learning: The SLiM framework unifies masked feature prediction and contrastive learning via a shared encoder to break reliance on dense tokenization, enabling effective learning with a highly compact token grid. This structural change reduces inference computation by 7.89 times compared to dense-token MAE baselines.

  2. Decoder-Free Masked Feature Prediction: SLiM reformulates the objective as a decoder-free, teacher-guided feature prediction task, shifting the target from low-level coordinates to highlevel semantics, which completely eliminates the need for dense tokenization and heavy decoders.

  3. Semantic Tube Masking for Shortcut Prevention: By introducing Semantic Tube Masking, the system occludes connected anatomical regions over consecutive frames to prevent trivial shortcut learning arising from strong inter-joint dependencies of human, ensuring deep skeletal-temporal reasoning and anatomical consistency.

  4. Skeleton-Aware Augmentations for Anatomical Validity: The system utilizes Skeleton-Aware Augmentations such as skeleton-aware rotation, mirroring, and bone-aware scaling to generate diverse views while strictly preserving anatomical validity through geometric transformations that maintain the articulated structure of the human body.

  5. Enhanced Representation Robustness: The system demonstrates robustness to declared missingness by routing unavailable joints through a learned mask token, and its performance is improved when pre-training data includes unflagged zero-fill or Gaussian noise, providing a mechanism for handling certain forms of corruption.

Abstract

Current skeleton representation learning paradigms face distinct limitations: Contrastive Learning (CL) often overlooks fine-grained motion details, while Masked Auto-Encoders (MAE) rely on coordinate-level reconstruction. This reconstruction inherently demands dense token sequences and heavy decoders, wasting pre-training computation on discarded components and forcing downstream inference to process dense token grids. To resolve these bottlenecks, we propose SLiM (Skeleton Less is More), a compact-token framework that unifies masked feature prediction and contrastive learning via a shared encoder. By shifting the objective from raw coordinate reconstruction to decoder-free, teacher-guided feature prediction, SLiM breaks the reliance on dense tokenization and enables effective learning with a highly compact token grid. Crucially, to prevent trivial shortcut learning arising from strong inter-joint dependencies of human, we introduce Semantic Tube Masking together with Skeleton-Aware Augmentations to enforce deep skeletal-temporal reasoning and anatomical consistency. Extensive experiments across multiple downstream protocols demonstrate that SLiM achieves state-of-the-art performance while structurally reducing inference computation by 7.89 times compared to dense-token MAE baselines.

Sources

Related papers