UniverSat: Resolution- and Modality-Agnostic Transformers for Earth Observation
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: I'm Tom.
Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.
Tom: We're looking at the paper: "UniverSat: Resolution- and Modality-Agnostic Transformers for Earth Observation".
Jane: Here's the gist of the paper. Vision Transformers (ViT) dominate computer vision; however, their reliance on rigid patch projectors hinders transfer to Earth Observation (EO), where input modalities, scales, and resolutions vary widely <ref:2606.23503#pg2> The paper introduces UniverSat, a ViT-style backbone built around a Universal Patch Encoder that maps patches from arbitrary spatial, spectral, and temporal resolutions into a shared embedding space with shared weights <ref:2606.23503#pg2> This enables training a single model on heterogeneous multimodal corpora via self-supervision, yielding robust, sensor-agnostic spatial features <ref:2606.23503#pg2>.
Tom: The Core Innovation of the Architecture The central innovation is the Universal Patch Encoder (UPE), which replaces fixed patch projectors with a UPE that uses linear-complexity axial cross-attention to embed patches of arbitrary spatial, spectral, and temporal dimensions into a shared latent space <ref:2606.23503#pg2>.
Jane: This design provides four key advantages for the model <ref:2606.23503#pg2>: 1.
Tom: Modality-Agnostic: A single set of weights processes many modalities combinations and arbitrary resolutions without input resampling or channel filtering <ref:2606.23503#pg2>.
Paper summary: Tom: So, we’re looking at this paper called UniverSat, and basically, it’s a Vision Transformer backbone. But the thing is that regular ViTs are really rigid; they rely on these fixed patch projectors, which makes them hard to move from standard vision tasks over to Earth Observation data.
Jane: Right. And what UniverSat tries to do is introduce this Universal Patch Encoder that maps patches from any spatial, spectral, and temporal resolutions into one shared embedding space with shared weights. That’s the core idea they are pushing here <ref:2606.23503#pg2>.
Lu: The main claim is that you can train just one model on a huge mix of different data types—heterogeneous multimodal corpora—using self-supervision, and this yields these robust, sensor-agnostic spatial features <ref:2606.23503#pg2>.
Meng: So, if I'm getting that, it means we don't have to build a whole new model every time we switch from Sentinel data to radar or some hyperspectral imagery.
Tom: Exactly. The innovation is in the Universal Patch Encoder itself <ref:2606.23503#pg2>. They use something called axial cross-attention with linear complexity to embed patches regardless of their input dimensions, spatial, spectral, and temporal size <ref:2606.23503#pg2>.
Jane: That sounds like it could solve the problem of having to resample or filter data just to fit a standard architecture.
Lu: It also gives you four specific advantages for the model: it’s modality-agnostic, resolution-flexible, granular in its skip connection, and it stays compatible with standard self-supervised learning objectives <ref:2606.23503#pg2>.
Meng: I’m interested in the resolution flexibility. Does that mean we don't have to decide on the output feature map size before we even start training?
Tom: It does. The spatial resolution of the output feature map is specified at inference time, and it's decoupled from the input patch size <ref:2606.23503#pg2>. That’s a practical thing for deployment.
Jane: And how do they train this thing up? How do you get all those modalities to learn together?
Lu: They use a self-supervised objective that builds on latent masked modeling, but they add scale augmentation and masking—they randomly drop about ninety percent of input atoms for each tile <ref:2606.23503#pg2>.
Meng: Ninety percent is aggressive. How do you keep the model stable with that much missing information?
Tom: They couple that up with cross-modal contrast, a batch-wise loss called Lcon, to make sure the embeddings for different visible patches are modality-invariant <ref:2606.23503#pg2>.
Jane: And there’s also this linear multimodal masked modeling, LM3, which predicts masked patches at specific target timestamps using those visible embeddings <ref:2606.23503#pg2>.
Lu: It sounds like they are harmonizing all this information into a single shared latent space, and that’s what allows the model to handle unseen sensors without needing input resampling <ref:2606.23503#pg2>.
Conclusion: Tom: So, looking at UniverSat, the title says it’s about resolution- and modality-agnostic transformers for Earth Observation <ref:2606.23503#pg1>. The authors are Yohann Perron, Guillaume Astruc, Nicolas Gonthier, Clément Mallet, Loic Landrieu <ref:2606.23503#pg1>.
Jane: What does that really mean for us? In simple terms, it means we’ve built a backbone that can handle the massive variety of inputs in Earth Observation—spatial size, different sensor types like optical and radar, and time series—without forcing us to clean or change the data beforehand.
Lu: The implication is that instead of building specialized models for every sensor combination you might want to use, you can train this one model once on a diverse set of data <ref:2606.23503#pg1>.
Meng: It trades specialization for generality, which means there's some overhead in the architecture itself that we have to deal with if the data isn't super complex.
Tom: That overhead is what you pay for when you can handle heterogeneous, multimodal data without input resampling <ref:2606.23503#pg1>. The real gain comes from generalizing to unseen sensors and modality combinations <ref:2606.23503#pg1>.
Jane: It’s about getting the model to be robust across scales, spectral bands, and timeframes simultaneously, all using just one set of weights <ref:2606.23503#pg1>. That capability is what makes this architecture so interesting for foundation models in this field.
Yohann Perron, Guillaume Astruc, Nicolas Gonthier, Clément Mallet Loic Landrieu
LIGM Ecole Nationale des Ponts et Chaussées IP Paris University Gustave Eiffel CNRS LASTIG IGN CNES EFEO
cs.CV
Submitted: 2026-06-22
Updated: 2026-10-29
Comments: NeurIPS 2026
Code: https://github.com/gastruc/UniverSat
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 78/100
The gist: Vision Transformers (ViT) dominate computer vision; however, their reliance on rigid patch projectors hinders transfer to Earth Observation (EO), where input modalities, scales…
Key concepts
- Universal Patch Encoder (UPE)
- This is the core innovation that replaces fixed patch projectors. The UPE uses linear-complexity axial cross-attention to embed input patches of any spatial, spectral, and temporal dimensions into a shared latent space. This design ensures the model can process various data types without needing specific pre-processing for each modality.
- Axial Cross-Attention (ACA)
- A specific attention mechanism used in the UPE that restricts attention to a single axis. This restriction is crucial because it guarantees linear complexity in the number of atomic tokens, meaning the computational cost scales efficiently regardless of how many dimensions (spatial, spectral, temporal) are involved.
- Modality-Agnostic Training
- The model is trained using self-supervised objectives that incorporate scale augmentation and cross-modal contrastive loss. This forces the shared latent space to learn representations that are invariant to the specific sensor or modality used during training, leading to robust, sensor-agnostic spatial features.
- Resolution Flexibility
- The architecture decouples the output feature map's spatial resolution from the input patch size. This means researchers can specify the desired output resolution at inference time, making it highly flexible for generalizing predictions to unseen datasets without requiring input resampling.
Terminology
Summary
Vision Transformers (ViT) dominate computer vision; however, their reliance on rigid patch projectors hinders transfer to Earth Observation (EO), where input modalities, scales, and resolutions vary widely. p. 0 UniverSat: Resolution- and Modality-Agnostic
Transformers for Earth Observation
Yohann Perron˚,1,5 Guillaume Astruc˚,1,2,4 Nicolas Gonthier2,3
Clément Mallet2 Loic Landrieu1,2
1LIGM, Ecole Nationale des Ponts et Chaussées, IP Paris, Univ Gustave Eiffel, CNRS
2LASTIG, Univ Gustave Eiffel, IGN, ENSG, 3
IGN 4CNES 5EFEO
Abstract
Vision Transformers (ViT) dominate computer vision. However, their reliance on rigid
patch projectors hinders transfer to Earth Observation (EO), where input modalities, scales,
and resolutions vary widely. However, their reliance on rigid
patch projectors hinders transfer to Earth Observation (EO), where input modalities, scales,
and resolutions vary widely. We introduce UniverSat, a ViT-style backbone built around a
Universal Patch Encoder that maps patches from arbitrary spatial, spectral, and temporal
resolutions, and from both optical and non-optical sensors, into a shared embedding space
with a shared set of weights. We introduce UniverSat, a ViT-style backbone built around a
Universal Patch Encoder that maps patches from arbitrary spatial, spectral, and temporal
resolutions, and from both optical and non-optical sensors, into a shared embedding space
with a shared set of weights. This enables training a single model on heterogeneous
multimodal corpora via self-supervision, yielding robust, sensor-agnostic spatial features.Where the paper says it
Where the paper says it
Where the paper says it
The Core Innovation
UNIVERSAT replaces fixed patch projectors with a Universal Patch Encoder (UPE) that uses linear-complexity axial cross-attention to embed patches of arbitrary spatial, spectral, and temporal dimensions into a shared latent space. This design provides four key advantages:
-
Modality-Agnostic; a single set of weights processes many modalities combinations and arbitrary resolutions without input resampling or channel filtering. p. 0
Where the paper says it
Integrated into a transformer operating over spatialized tokens, this design provides four key advantages: • Modality-Agnostic. A single set of weights processes many modalities combinations and arbitrary resolutions without input resampling or channel filtering.
-
Resolution-flexible; the spatial resolution of the output feature map is specified at inference time and decoupled from the input patch size. p. 1
Where the paper says it
resolution TerraMind [9] 1 1 1 ✓ OmniSat [12] 2 2 1 1 ✓ EarthView [23] 3 1 1 ✓ Panopticon [13] 5 1 1 ✓ Galileo [6] 1 1 1 ✓ ✓ AnySat [7] 2 4 2 1 ✓ ✓ DOFA [16] 3 1 1 ✓ ✓ FoMo [24] 4 2 1 1 ✓ ✓ Ramen [17] 1 1 1 1 ✓ ✓ ✓ UniverSat (ours) 4 4 2 2 3 ✓ ✓ ✓ : optical snapshot Å : optical time series : radar : elevation : hyperspectral output resolutions: : image : patch : pixel : selectable • Resolution-flexible. The spatial resolution of the output feature map is specified at inference time and decoupled from the input patch size.
-
Granular; a skip connection preserves fine spatial details beyond patch-level embeddings. p. 1
Where the paper says it
• Granular. A skip connection preserves fine spatial details beyond patch-level embeddings.
-
SSL-compatible; the architecture retains the core ViT structure, enabling training with standard self-supervised learning objectives. p. 1
Where the paper says it
• SSL-compatible. The architecture retains the core ViT structure, enabling training with standard self-supervised learning objectives.
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Universal Patch Encoder (UPE) allows a single model to process patches from
arbitrary spatial, spectral, and temporal resolutions,
enabling training on heterogeneous multimodal corpora via self-supervision without input resampling or channel filtering. This enables the system to handle inputs ranging from 0.1 m to 300 m spatial resolutions and 1 to 396 spectral channels simultaneously. -
Modality-Agnostic Training Enables Cross-Sensor Transfer A single set of weights processes
many modalities combinations and arbitrary resolutions without input resampling or channel filtering,
meaning the resulting model can generalize to unseen sensors within this gamut without requiring modality-specific projectors. -
Resolution-Flexible Output Enables Fine Detail Recovery The system allows the
spatial resolution of the output feature map is specified at inference time and decoupled from the input patch size,
while preserving fine spatial details through askip connection
that attends to sub-patch embeddings. -
Multimodal Self-Supervised Training Trains on Heterogeneous Data The training framework combines
cross-modal contrast at the patch level and latent multimodal masked modeling (LM3),
allowing a single model to be trained jointly on 13 sensors across four modalities, yielding robust, sensor-agnostic spatial features. -
Any-Resolution Prediction Enables Ground Sampling Distance Control Users can
specify a target ground sampling distance (GSD), defining a new output spatial grid O,
and recover fine details by having the target tokenattend to all sub-patch embeddings via cross-attention CA.
-
Linear Multimodal Masked Modeling (LM3) Captures Temporal and Spectral Contexts The LM3 objective predicts masked patches at selected timestamps using learned mask tokens, while using a contrastive loss to ensure that the shared latent space preserves
information from all modalities in a shared latent space.
Abstract
Vision Transformers (ViT) dominate computer vision. However, their reliance on rigid patch projectors hinders transfer to Earth Observation (EO), where input modalities, scales, and resolutions vary widely. We introduce UniverSat, a ViT-style backbone built around a Universal Patch Encoder that maps patches from arbitrary spatial, spectral, and temporal resolutions, and from both optical and non-optical sensors, into a shared embedding space with a shared set of weights. This enables training a single model on heterogeneous multimodal corpora via self-supervision, yielding robust, sensor-agnostic spatial features. We validate this approach with strong results across classification and segmentation on standard EO benchmarks from GeoBench, PANGEABench, and SpectralEarth. Our code and models are available at https://github.com/gastruc/UniverSat.
Sources
- TerraMind: Large-Scale Generative Multimodality for Earth Observation
- AlphaEarth Foundations: An embedding field model for accurate and efficient global mapping from sparse label data
- Panopticon: Advancing Any-Sensor Foundation Models for Earth Observation
- SenPa-MAE: Sensor Parameter Aware Masked Autoencoder for Multi-Satellite Self-Supervised Pretraining
- Neural Plasticity-Inspired Multimodal Foundation Model for Earth Observation
- RAMEN: Resolution-Adjustable Multimodal Encoder for Earth Observation
- PANGAEA: A Global and Inclusive Benchmark for Geospatial Foundation Models
- TESSERA: Temporal Embeddings of Surface Spectra for Earth Representation and Analysis
- Lightweight, Pre-trained Transformers for Remote Sensing Timeseries
- TerraFM: A Scalable Foundation Model for Unified Multisensor Earth Observation
- USat: A Unified Self-Supervised Encoder for Multi-Sensor Satellite Imagery
- PyViT-FUSE: A Foundation Model for Multi-Sensor Earth Observation Data
- Axial Attention in Multidimensional Transformers
- Zooming Out on Zooming In: Advancing Super-Resolution for Remote Sensing
- DINOv3
- Towards a Unified Copernicus Foundation Model for Earth Vision
- SpectralEarth: Training Hyperspectral Foundation Models at Scale
- Any Model, Any Place, Any Time: Get Remote Sensing Foundation Model Embeddings On Demand
- Cluster and Predict Latent Patches for Improved Masked Image Modeling
- Prithvi-EO-2.0: A Versatile Multi-Temporal Foundation Model for Earth Observation Applications
Related papers
- TEA: Text Encoder Alignment for Robust Concept Erasure in Text-to-Image Models
- Do Egocentric Video-Language Models Capture Both Hand- and Object-Centric Cues?
- Aura: Consistent Multi-Subject Video Generation via VLM-Grounded Semantic Alignment
- PVRF: All-in-one Adverse Weather Removal via Prior-modulated and Velocity-constrained Rectified Flow
- DLWM: Diverse Latent World Models for Efficient Multimodal Reasoning
- Out of Sight, Still in Mind: Token Compression for Omni-LLMs