W2Rep: Learning Visual Representations by Watching the World Change
cs.CV
Submitted: 2026-09-28
Updated: 2026-09-28
Project page: https://wenooi.github.io/W2Rep
License: http://creativecommons.org/licenses/by/4.0/
The gist: Images capture the world at one moment, whereas video reveals how it changes.
Terminology
Abstract
Images capture the world at one moment, whereas video reveals how it changes. Image self-supervision learns spatial structure from a single moment, while video methods commonly learn temporal relationships inside a representation computed jointly from several frames. We ask whether watching a scene change can instead improve features available from one image without sacrificing the ability to represent video. We introduce W2Rep, a masked feature-prediction framework in which an independently encoded source image participates in prediction at the same or another moment. The predictor is conditioned on visible video context, the queried location, and the signed time interval between source and target. This gives the cross-frame objective two complementary roles: the image path learns features that remain useful across time, while the video path must gather evidence that is missing from the source image. Across model scales and downstream tasks, W2Rep improves frozen and fine-tuned recognition under our comparison protocol, while joint video encoding provides further gains over frame-wise aggregation. Controlled experiments show that these gains depend on directly updating the source-image features and on using both video context and temporal displacement. Overall, change across a video can supervise a visual encoder whose representations remain useful at either image or video granularity. Code is available at https://wenooi.github.io/W2Rep.
Sources
- BEiT: BERT Pre-Training of Image Transformers
- VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning
- MC-JEPA: A Joint-Embedding Predictive Architecture for Self-Supervised Learning of Motion and Content Features
- Revisiting Feature Prediction for Learning Visual Representations from Video
- You Don't Need Strong Assumptions: Visual Representation Learning via Temporal Differences
- Visual Representation Learning with Stochastic Frame Prediction
- Decoupled Weight Decay Regularization
- UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
- iBOT: Image BERT Pre-Training with Online Tokenizer
Related papers
- Loss Knows Best: Detecting Annotation Errors in Videos via Loss Trajectories
- AnchorWeave: World-Consistent Video Generation with Retrieved Local Spatial Memories
- Benchmarking the Robustness of Foundation Models for Mammography under Domain Shift
- MambaX-Net: Dual-Input Mamba-Enhanced Cross-Attention Network for Longitudinal MRI Segmentation
- TeleOCR: Navigating Document Parsing Across Digital and Camera-Captured Documents
- A Survey on Efficient Vision-Language-Action Models