UniverSat: Resolution- and Modality-Agnostic Transformers for Earth Observation
summary
The gist
Vision Transformers (ViT) dominate computer vision; however, their reliance on rigid patch projectors hinders transfer to Earth Observation (EO), where input modalities, scales…
In short
UniverSat is a Vision Transformer backbone designed for Earth Observation that handles diverse inputs like different sensors, resolutions, and modalities without needing input resampling. It uses a Universal Patch Encoder to map arbitrary spatial, spectral, and temporal patches into a single shared embedding space. This allows training one model on many heterogeneous datasets.
Key concepts
- Universal Patch Encoder (UPE)
- This is the core innovation that replaces fixed patch projectors. The UPE uses linear-complexity axial cross-attention to embed input patches of any spatial, spectral, and temporal dimensions into a shared latent space. This design ensures the model can process various data types without needing specific pre-processing for each modality.
- Axial Cross-Attention (ACA)
- A specific attention mechanism used in the UPE that restricts attention to a single axis. This restriction is crucial because it guarantees linear complexity in the number of atomic tokens, meaning the computational cost scales efficiently regardless of how many dimensions (spatial, spectral, temporal) are involved.
- Modality-Agnostic Training
- The model is trained using self-supervised objectives that incorporate scale augmentation and cross-modal contrastive loss. This forces the shared latent space to learn representations that are invariant to the specific sensor or modality used during training, leading to robust, sensor-agnostic spatial features.
- Resolution Flexibility
- The architecture decouples the output feature map's spatial resolution from the input patch size. This means researchers can specify the desired output resolution at inference time, making it highly flexible for generalizing predictions to unseen datasets without requiring input resampling.
Terminology used across episodes
This episode discusses
- UniverSat: Resolution- and Modality-Agnostic Transformers for Earth Observation · Paper Radio
- TerraMind: Large-Scale Generative Multimodality for Earth Observation
- AlphaEarth Foundations: An embedding field model for accurate and efficient global mapping from sparse label data
- Panopticon: Advancing Any-Sensor Foundation Models for Earth Observation
- SenPa-MAE: Sensor Parameter Aware Masked Autoencoder for Multi-Satellite Self-Supervised Pretraining
- Neural Plasticity-Inspired Multimodal Foundation Model for Earth Observation
- RAMEN: Resolution-Adjustable Multimodal Encoder for Earth Observation
- PANGAEA: A Global and Inclusive Benchmark for Geospatial Foundation Models
- TESSERA: Temporal Embeddings of Surface Spectra for Earth Representation and Analysis
- Lightweight, Pre-trained Transformers for Remote Sensing Timeseries
- TerraFM: A Scalable Foundation Model for Unified Multisensor Earth Observation
- USat: A Unified Self-Supervised Encoder for Multi-Sensor Satellite Imagery
- PyViT-FUSE: A Foundation Model for Multi-Sensor Earth Observation Data
- Axial Attention in Multidimensional Transformers
- Zooming Out on Zooming In: Advancing Super-Resolution for Remote Sensing
- DINOv3
- Towards a Unified Copernicus Foundation Model for Earth Vision
- SpectralEarth: Training Hyperspectral Foundation Models at Scale
- Any Model, Any Place, Any Time: Get Remote Sensing Foundation Model Embeddings On Demand
- Cluster and Predict Latent Patches for Improved Masked Image Modeling
- Prithvi-EO-2.0: A Versatile Multi-Temporal Foundation Model for Earth Observation Applications
The paper
UniverSat: Resolution- and Modality-Agnostic Transformers for Earth Observation · Read on arXiv
Yohann Perron, Guillaume Astruc, Nicolas Gonthier, Clément Mallet Loic Landrieu
LIGM Ecole Nationale des Ponts et Chaussées IP Paris University Gustave Eiffel CNRS LASTIG IGN CNES EFEO
Vision Transformers (ViT) dominate computer vision. However, their reliance on rigid patch projectors hinders transfer to Earth Observation (EO), where input modalities, scales, and resolutions vary widely. We introduce UniverSat, a ViT-style backbone built around a Universal Patch Encoder that maps patches from arbitrary spatial, spectral, and temporal resolutions, and from both optical and non-optical sensors, into a shared embedding space with a shared set of weights. This enables training a single model on heterogeneous multimodal corpora via self-supervision, yielding robust, sensor-agnostic spatial features. We validate this approach with strong results across classification and segmentation on standard EO benchmarks from GeoBench, PANGEABench, and SpectralEarth. Our code and models are available at https://github.com/gastruc/UniverSat.
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: I'm Tom.
Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.
Tom: We're looking at the paper: "UniverSat: Resolution- and Modality-Agnostic Transformers for Earth Observation".
Jane: Here's the gist of the paper. Vision Transformers (ViT) dominate computer vision; however, their reliance on rigid patch projectors hinders transfer to Earth Observation (EO), where input modalities, scales, and resolutions vary widely <ref:2606.23503#pg2> The paper introduces UniverSat, a ViT-style backbone built around a Universal Patch Encoder that maps patches from arbitrary spatial, spectral, and temporal resolutions into a shared embedding space with shared weights <ref:2606.23503#pg2> This enables training a single model on heterogeneous multimodal corpora via self-supervision, yielding robust, sensor-agnostic spatial features <ref:2606.23503#pg2>.
Tom: The Core Innovation of the Architecture The central innovation is the Universal Patch Encoder (UPE), which replaces fixed patch projectors with a UPE that uses linear-complexity axial cross-attention to embed patches of arbitrary spatial, spectral, and temporal dimensions into a shared latent space <ref:2606.23503#pg2>.
Jane: This design provides four key advantages for the model <ref:2606.23503#pg2>: 1.
Tom: Modality-Agnostic: A single set of weights processes many modalities combinations and arbitrary resolutions without input resampling or channel filtering <ref:2606.23503#pg2>.
Paper summary: Tom: So, we’re looking at this paper called UniverSat, and basically, it’s a Vision Transformer backbone. But the thing is that regular ViTs are really rigid; they rely on these fixed patch projectors, which makes them hard to move from standard vision tasks over to Earth Observation data.
Jane: Right. And what UniverSat tries to do is introduce this Universal Patch Encoder that maps patches from any spatial, spectral, and temporal resolutions into one shared embedding space with shared weights. That’s the core idea they are pushing here <ref:2606.23503#pg2>.
Lu: The main claim is that you can train just one model on a huge mix of different data types—heterogeneous multimodal corpora—using self-supervision, and this yields these robust, sensor-agnostic spatial features <ref:2606.23503#pg2>.
Meng: So, if I'm getting that, it means we don't have to build a whole new model every time we switch from Sentinel data to radar or some hyperspectral imagery.
Tom: Exactly. The innovation is in the Universal Patch Encoder itself <ref:2606.23503#pg2>. They use something called axial cross-attention with linear complexity to embed patches regardless of their input dimensions, spatial, spectral, and temporal size <ref:2606.23503#pg2>.
Jane: That sounds like it could solve the problem of having to resample or filter data just to fit a standard architecture.
Lu: It also gives you four specific advantages for the model: it’s modality-agnostic, resolution-flexible, granular in its skip connection, and it stays compatible with standard self-supervised learning objectives <ref:2606.23503#pg2>.
Meng: I’m interested in the resolution flexibility. Does that mean we don't have to decide on the output feature map size before we even start training?
Tom: It does. The spatial resolution of the output feature map is specified at inference time, and it's decoupled from the input patch size <ref:2606.23503#pg2>. That’s a practical thing for deployment.
Jane: And how do they train this thing up? How do you get all those modalities to learn together?
Lu: They use a self-supervised objective that builds on latent masked modeling, but they add scale augmentation and masking—they randomly drop about ninety percent of input atoms for each tile <ref:2606.23503#pg2>.
Meng: Ninety percent is aggressive. How do you keep the model stable with that much missing information?
Tom: They couple that up with cross-modal contrast, a batch-wise loss called Lcon, to make sure the embeddings for different visible patches are modality-invariant <ref:2606.23503#pg2>.
Jane: And there’s also this linear multimodal masked modeling, LM3, which predicts masked patches at specific target timestamps using those visible embeddings <ref:2606.23503#pg2>.
Lu: It sounds like they are harmonizing all this information into a single shared latent space, and that’s what allows the model to handle unseen sensors without needing input resampling <ref:2606.23503#pg2>.
Conclusion: Tom: So, looking at UniverSat, the title says it’s about resolution- and modality-agnostic transformers for Earth Observation <ref:2606.23503#pg1>. The authors are Yohann Perron, Guillaume Astruc, Nicolas Gonthier, Clément Mallet, Loic Landrieu <ref:2606.23503#pg1>.
Jane: What does that really mean for us? In simple terms, it means we’ve built a backbone that can handle the massive variety of inputs in Earth Observation—spatial size, different sensor types like optical and radar, and time series—without forcing us to clean or change the data beforehand.
Lu: The implication is that instead of building specialized models for every sensor combination you might want to use, you can train this one model once on a diverse set of data <ref:2606.23503#pg1>.
Meng: It trades specialization for generality, which means there's some overhead in the architecture itself that we have to deal with if the data isn't super complex.
Tom: That overhead is what you pay for when you can handle heterogeneous, multimodal data without input resampling <ref:2606.23503#pg1>. The real gain comes from generalizing to unseen sensors and modality combinations <ref:2606.23503#pg1>.
Jane: It’s about getting the model to be robust across scales, spectral bands, and timeframes simultaneously, all using just one set of weights <ref:2606.23503#pg1>. That capability is what makes this architecture so interesting for foundation models in this field.
More episodes
- 2603.28595-Optimistic Actor-Critic with Parametric Policies for Linear Markov Decision Processes
- 2603.29765-Training-Free Dynamic Upcycling of Expert Language Models
- 2605.00778-Observable Performance Does Not Fully Reflect Adaptive System Organization: A Multi-Level Analysis of Gait Dynamics Under Occlusal Constraint
- 2605.06906-TraXion: Rethinking Pre-training Frameworks for Mobility and Beyond
- 2605.09165-Sparse Layers are Critical to Scaling Looped Language Models
- 2605.09439-Inverse Design for Conditional Distribution Matching
- 2605.10313-Signature Approach for Contextual Bandits with Nonlinear and Path-dependent Rewards: Regret Bound and Statistical Inference
- 2605.10878-Minimum Neural Weight Norm = Kolmogorov Complexity
- 2605.11617-MIST: Reliable Streaming Decision Trees for Online Class-Incremental Learning via McDiarmid Bound
- 2605.15085-From Data to Action: Accelerating Refinery Optimization with AI