Beyond Local Linearity: Scale-Resolved Geometry of Learned Image Encoders
cs.CV
Submitted: 2026-09-30
Updated: 2026-09-30
Comments: Extended abstract, NeurIPS 2026 Workshop on Symmetry and Geometry in Neural Representations (NeurReps). 14 pages, 5 figures
License: http://creativecommons.org/licenses/by/4.0/
The gist: Understanding how learned representations respond to finite input changes is important for characterizing their sensitivity, invariances, and robustness.
Terminology
Abstract
Understanding how learned representations respond to finite input changes is important for characterizing their sensitivity, invariances, and robustness. Yet existing geometric analyses are predominantly local and describe only infinitesimal perturbations. We introduce a scale-resolved statistic that compares an encoder's measured feature displacement with its local linear prediction as the perturbation magnitude increases. Across diverse image encoders, we discover a characteristic plateau-rise-peak-decay profile, which we call the bump. The bump is absent at initialization, emerges early during standard training, and does not form under randomized labels or random-noise inputs. Its shape also varies with the training distribution and robustness objective. These results establish departures from local geometry as a signature of how encoder representations are shaped by learning.
Sources
- On Lazy Training in Differentiable Programming
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- Discriminating image representations with principal distortions
- Complexity of Linear Regions in Deep Networks
- Deep ReLU Networks Have Surprisingly Few Activation Patterns
- Deep Residual Learning for Image Recognition
- Deep Networks Always Grok and Here is Why
- Neural Tangent Kernel: Convergence and Generalization in Neural Networks
- A ConvNet for the 2020s
- Geometric Autoencoders -- What You See is What You Decode
- Sensitivity and Generalization in Neural Networks: an Empirical Study
- Exponential expressivity in deep neural networks through transient chaos
- Adversarial Robustness through Local Linearization
- On the Expressive Power of Deep Neural Networks
- ImageNet Large Scale Visual Recognition Challenge
- Fake it till you make it: Learning transferable representations from synthetic ImageNet clones
- The Riemannian Geometry of Deep Generative Models
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Metrics for Probabilistic Geometries
- Wide Residual Networks
Related papers
- Loss Knows Best: Detecting Annotation Errors in Videos via Loss Trajectories
- AnchorWeave: World-Consistent Video Generation with Retrieved Local Spatial Memories
- Benchmarking the Robustness of Foundation Models for Mammography under Domain Shift
- MambaX-Net: Dual-Input Mamba-Enhanced Cross-Attention Network for Longitudinal MRI Segmentation
- TeleOCR: Navigating Document Parsing Across Digital and Camera-Captured Documents
- A Survey on Efficient Vision-Language-Action Models