Self-Supervised Learning as Discrete Communication
cs.CV, cs.IR, cs.LG
Submitted: 2026-02-10
Updated: 2026-06-15
Journal ref: Proceedings of the 43rd International Conference on Machine Learning, Proceedings of Machine Learning Research, 306, 152937-152956, 2026
Code: https://github.com/fastai/imagenette
License: http://creativecommons.org/licenses/by/4.0/
The gist: Most self-supervised learning (SSL) methods learn continuous visual representations by aligning different views of the same input…
Terminology
Abstract
Most self-supervised learning (SSL) methods learn continuous visual representations by aligning different views of the same input, offering limited control over how information is structured across representation dimensions. In this work, we frame visual self-supervised learning as a discrete communication process between a teacher and a student network, where semantic information is transmitted through a fixed-capacity binary channel. Rather than aligning continuous features, the student predicts multi-label binary messages produced by the teacher. Discrete agreement is enforced through an element-wise binary cross-entropy objective, while a coding-rate regularization term encourages effective utilization of the constrained channel, promoting structured representations. We further show that periodically reinitializing the projection head strengthens this effect by encouraging embeddings that remain predictive across multiple discrete encodings. Extensive experiments demonstrate consistent improvements over continuous agreement baselines on image classification, retrieval, and dense visual prediction tasks, as well as under domain shift through self-supervised adaptation. Beyond backbone representations, we analyze the learned binary codes and show that they form a compact and informative discrete language, capturing semantic factors reusable across classes.
Sources
- BEiT: BERT Pre-Training of Image Transformers
- VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning
- Scalable Pre-training of Large Autoregressive Image Models
- ViT2Hash: Unsupervised Information-Preserving Hashing
- Learning deep representations by mutual information estimation and maximization
- Neural Manifold Clustering and Embedding
- Image Hashing via Cross-View Code Alignment in the Age of Foundation Models
- DINOv2: Learning Robust Visual Features without Supervision
- The 2017 DAVIS Challenge on Video Object Segmentation
- DINOv3
- EMP-SSL: Towards Self-Supervised Learning in One Training Epoch
- Joint Embedding vs Reconstruction: Provable Benefits of Latent Space Prediction for Self Supervised Learning
- Simplifying DINO via Coding Rate Regularization
- Next-Embedding Prediction Makes Strong Vision Learners
- In Pursuit of Pixel Supervision for Visual Pre-training
- Learning Diverse and Discriminative Representations via the Principle of Maximal Coding Rate Reduction
- iBOT: Image BERT Pre-Training with Online Tokenizer
Related papers
- TEA: Text Encoder Alignment for Robust Concept Erasure in Text-to-Image Models
- Do Egocentric Video-Language Models Capture Both Hand- and Object-Centric Cues?
- Aura: Consistent Multi-Subject Video Generation via VLM-Grounded Semantic Alignment
- PVRF: All-in-one Adverse Weather Removal via Prior-modulated and Velocity-constrained Rectified Flow
- DLWM: Diverse Latent World Models for Efficient Multimodal Reasoning
- Out of Sight, Still in Mind: Token Compression for Omni-LLMs