Three Necessary Principles for Self-Supervised Visual Representation Learning
cs.CV, cs.AI, cs.LG
Submitted: 2026-08-08
Updated: 2026-09-02
Comments: Accepted at the 19th European Conference on Computer Vision (ECCV 2026) Workshops
License: http://creativecommons.org/licenses/by/4.0/
The gist: We argue that learning visual representations without labels requires a training signal jointly complete across three non-overlapping objectives: semantic invariance across augmented views,
Terminology
Abstract
We argue that learning visual representations without labels requires a training signal jointly complete across three non-overlapping objectives: semantic invariance across augmented views, patch-level spatial prediction, and representational non-degeneracy. We formalize these as the observation, prediction, and regularization principles and prove (i) that combining observation and prediction without regularization admits the constant encoder as a global minimizer under negative-free alignment; (ii) that the two objectives are gradient-complementary and structurally non-conflicting at the encoder output; and (iii) that the momentum encoder converges to the same fixed point as the online encoder and provides no collapse guarantee at convergence. Contrastive alignment provides only self-limiting collapse resistance, formalized via an explicit gradient-decay argument. Dropping prediction withholds the spatial training signal by construction; dropping observation forfeits cross-view semantic invariance by construction; at the scale we study, no pair substitutes for the third. Every major self-supervised method is a special case of a single unified energy decomposition. We pair every theoretical claim with a controlled experiment, including a patch-retrieval evaluation for the spatial consequence of prediction.
Sources
- V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning
- LeJEPA: Provable and Scalable Self-Supervised Learning Without the Heuristics
- Revisiting Feature Prediction for Learning Visual Representations from Video
- Introduction to Latent Variable Energy-Based Models: A Path Towards Autonomous Machine Intelligence
- On the duality between contrastive and non-contrastive self-supervised learning
- Relational Representation Distillation
- SynCo: Synthetic Hard Negatives for Contrastive Visual Representation Learning
- Masked Autoencoders Are Scalable Vision Learners
- Rectified LpJEPA: Joint-Embedding Predictive Architectures with Sparse and Maximum-Entropy Representations
- Representation Learning with Contrastive Predictive Coding
- Barlow Twins: Self-Supervised Learning via Redundancy Reduction
- iBOT: Image BERT Pre-Training with Online Tokenizer
Related papers
- Loss Knows Best: Detecting Annotation Errors in Videos via Loss Trajectories
- AnchorWeave: World-Consistent Video Generation with Retrieved Local Spatial Memories
- Benchmarking the Robustness of Foundation Models for Mammography under Domain Shift
- MambaX-Net: Dual-Input Mamba-Enhanced Cross-Attention Network for Longitudinal MRI Segmentation
- TeleOCR: Navigating Document Parsing Across Digital and Camera-Captured Documents
- A Survey on Efficient Vision-Language-Action Models