MEOX: Modality-Conditioned Routing Before Fusion in a Compact Earth Observation Encoder
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: We're looking at the paper: "MEOX: Modality-Conditioned Routing Before Fusion in a Compact Earth Observation Encoder".
Jane: Here's the gist of the paper. MEOX presents a compact multimodal masked autoencoder designed for Earth Observation that preserves modality-dependent processing before a learned patch-wise fusion The model achieves strong task transfer using a compact parameter budget.
Tom: Model Architecture and Components MEOX is a multimodal masked autoencoder with a 2.939 million-parameter encoder and 3.115 million parameters in total.
Jane: The architecture incorporates sensor-specific adapters, explicit validity signals, and a shared sparse-expert block to maintain modality-dependent processing before patch-wise fusion.
Tom: Four metadata tokens accompany a single spatial sequence through fourteen further encoder blocks.
Paper summary: Tom: So, looking at the MEOX paper and its title, Modality-Conditioned Routing Before Fusion in a Compact Earth Observation Encoder—it’s essentially about designing a smarter way for AI to look at messy, multi-sensor satellite data.
Jane: It tackles the issue of combining different sensor types by making sure each one gets its own focused processing path before they are merged. The authors show that this approach works well on things like six GEO-Bench tasks when tested at both sixty-four and two hundred twenty-four pixels, achieving results that beat some other models <ref:2609.05351#pg1,at both 64 and 224 pixels>.
Lu: What this means in simple terms is that instead of having one giant neural network trying to figure out how to combine all the satellite bands at once, you have a system that learns which sensor provides the best local context for every single pixel and fuses them based on those learned weights.
Meng: From a deployment standpoint, the compact parameter budget and the explicit handling of missing data suggest this is something we could actually deploy in operational settings without needing an enormous computational footprint just to get decent reconstruction results.
Lalam: I'm confident in what the literature points to; this approach provides a structured way for heterogeneous sensors to communicate their relevance before they ever get mixed together into the final output.
Tom: So, MEOX offers a compact architecture that uses sensor-specific adapters and explicit validity signals to maintain modality-dependent processing through an entire encoder, followed by a learned patch-wise fusion step. It’s about efficient routing before the heavy lifting of fusion.
Conclusion: Tom: So we've been looking at MEOX, which stands for Modality-Conditioned Routing Before Fusion in a Compact Earth Observation Encoder by the authors of this paper.
Jane: It basically shows how you can process different satellite images—like infrared and visible light—separately before you try to mix them together.
Lu: The core idea is using sensor-specific adapters and explicit validity signals to keep those modalities distinct throughout the encoding process.
Meng: That sounds like a lot of complexity if we're trying to ship something small and fast for production systems, you know?
Lalam: But they use a shared sparse expert block that keeps the parameter count down while still allowing that specialized processing before the fusion step.
Tom: So, they’ve managed to keep a pretty tight parameter budget while maintaining this modality-dependent flow.
Jane: And when we look at their results, they show strong task transfer using those compact parameters, which means it learns well across different tasks on the data.
Lu: They also use four extra metadata tokens alongside the spatial sequence—latitude, longitude, month, and climate context—to give the model more information about where and when it's looking.
Meng: That metadata stuff is interesting because it helps handle missing data differently; if you have a few pieces of that info missing, the contribution of that missing part to the final result just goes up.
Lalam: So, instead of filling in zeros everywhere, they’re letting the model know how much it needs to fill in based on what's actually available.
Tom: And they test it on six different GEO-Bench tasks at both sixty-four and two hundred twenty-four pixels, and those results beat some other models reported for this specific setup.
Jane: It suggests that this routing mechanism before fusion is a solid way to handle heterogeneous data in Earth observation without needing massive computational resources for every sensor combination.
Lu: The implication is that we can build more efficient encoders where the model intelligently weighs which sensor signal matters most at each exact spot on the image.
Meng: I wonder if those specific adapters are easy to fine-tune when we get new satellite data sources, or if they need a whole re-training process every time.
Lalam: They constrain parameter growth with these shared expert projections, which helps keep the model from just growing infinitely with every new sensor you throw at it.
Tom: It's about keeping the processing localized and focused on the specific sensor input before the final combination happens.
Jane: It’s a neat way to handle messy, multi-sensor data by making sure each piece gets its own thoughtful consideration before they merge into one final picture.
Lu: So, moving forward, we should look closely at how they handle those validity signals and see if that's a robust way to manage real-world sensor noise.
European Centre for Medium-Range Weather Forecasts
cs.CV
Submitted: 2026-09-04
Updated: 2026-11-03
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 90/100
The gist: MEOX presents a compact multimodal masked autoencoder designed for Earth Observation that preserves modality-dependent processing before a learned patch-wise fusion The model achieves strong task…
Key concepts
- Multimodal Masked Autoencoder
- This is an AI model designed to understand multiple types of input, like different satellite images, simultaneously. It learns to reconstruct missing parts of the input data by looking at all available pieces across different modalities (sensors) at once. It's a way for the model to learn how different data types relate to each other.
- Sensor-Specific Adapters
- These are small, specialized modules added within the main encoder that are tailored to handle the unique characteristics of each sensor. This allows the model to process data from one sensor in a way that is optimized for its specific format and noise, ensuring modality-dependent processing before any merging happens.
- Patch-wise Fusion Score
- This mechanism calculates a score for how important each available sensor's information is at every single location (patch) on the image. This score determines the weight given to each sensor's contribution during the final fusion step, ensuring that only relevant data influences the combined output.
- Sparse-Expert Block
- This is a shared component within the model that uses a sparse expert system. Instead of using every single parameter for every calculation, it selectively activates only a few specialized parts when needed. This keeps the total number of parameters small while still allowing the model to perform complex, modality-aware computations.
Terminology
Summary
MEOX presents a compact multimodal masked autoencoder designed for Earth Observation that preserves modality-dependent processing before a learned patch-wise fusion The model achieves strong task transfer using a compact parameter budget. p. 0 These results demonstrate sensor-flexible representation learning
and strong task transfer using a compact parameter budget. Index Terms—Earth observation, multimodal learning, masked
autoencoders, mixture of experts, remote sensing, compact representation learning.Where the paper says it
Model Architecture and Components
MEOX is a multimodal masked autoencoder with a 2.939 million-parameter encoder and 3. p. 0 We present
MEOX (Multimodal Earth Observation with eXperts), a multimodal masked autoencoder with a 2. 939 million-parameter
encoder and 3. 939 million-parameter
encoder and 3. 115 million parameters in total. 115 million parameters in total. Sensor-specific
adapters, explicit validity signals, and a shared sparse-expert
block preserve modality-dependent processing before a learned
patch-wise fusion. Sensor-specific
adapters, explicit validity signals, and a shared sparse-expert
block preserve modality-dependent processing before a learned
patch-wise fusion. Four metadata tokens then accompany a single
spatial sequence through fourteen further encoder blocks. Four metadata tokens then accompany a single
spatial sequence through fourteen further encoder blocks. Shared
expert projections with private low-rank residuals constrain
parameter growth, while rotary attention supports downstream
spatial grids different from pretraining.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Data Representation and Missingness Handling
Missing elements are numerically zero-filled only after validity is retained, meaning a missing value differs from a valid observation equal to its normalization mean. Each sensor image is divided into nonoverlapping P × P patches, resulting in N = HW/P2 spatial tokens. p. 1 Each sensor image is divided into nonoverlapping P × P
patches. An image of size H × W therefore produces N =
HW/P2
spatial tokens. With H = W = 64 and P = 4, each sensor produces
N = 16 × 16 = 256 tokens of width d = 144. Metadata is represented by four additional tokens, one
each for latitude, longitude, month, and ERA5 climate context. The first term projects the available values, ϵ
type
j
identifies
the metadata group, and ϵ
miss
j
represents missingness. The
missing-state contribution is zero when all components are
present and increases as more components become unavailable.Where the paper says it
Where the paper says it
Where the paper says it
Fusion Mechanism and Expert Routing
Before fusion, MEOX processes each sensor independently using the same transformer-MoE block, producing a representation zm = Fθ(tm). At each spatial position, the model computes one fusion score for every available sensor using Equation (3). p. 2 Attention and expert routing therefore
operate on sensor-specific observations before cross-sensor
fusion. At each spatial position, the model computes one fusion
score for every available sensor:
am,i = w
⊤
f LN(zm,i) + cm + logmax(qm,i, 10−6
)
, (3)
αm,i =
exp(am,i)
P
m′∈A exp(am′
,i)
, zi =
X
m∈A
αm,izm,i. (4)
Here, am,i is the unnormalized fusion score, wf ∈ R
d
is a
learned vector that scores the normalized patch representation, and cm is a learned sensor-specific bias. The quantity
qm,i ∈ [0, 1] is the fraction of valid elements in patch i of
sensor m. The lower bound 10−6 prevents numerical
evaluation of log(0). The softmax produces nonnegative fusion
weights αm,i that sum to one across the available sensors.Where the paper says it
Where the paper says it
Where the paper says it
Training Objectives and Evaluation
The pretraining objective balances modalities by valid target elements and reconstructs deliberately omitted sensors. The reconstruction loss for sensor m is calculated only on spatially masked and valid target values, defined by Equation (9). p. 3 The reconstruction loss is calculated only on spatially masked
and valid target values. Let m identify a sensor, n a sample
in the batch, i a spatial patch, and k an element within that
patch, including its pixel and channel position. Equal sensor averaging prevents this
channel-count difference from dominating training. The final
objective also includes an auxiliary term that encourages
balanced expert routing. 228 million MMEarth64 samples using modality-balanced
masked reconstruction and structured sensor dropout. Frozen
transfer is evaluated on six GEO-Bench tasks at both 64 and
224 pixels.Where the paper says it
Where the paper says it
Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Improved representation learning for Earth Observation by using MEOX, which is a multimodal masked autoencoder with 2.939 million parameters, enabling sensor-flexible representation learning and strong task transfer using a compact parameter budget.
-
Enhanced handling of missing data through explicit validity signals and metadata tokens, allowing the model to
distinguish unavailable values from valid normalized zeros
when processing Sentinel-1 or Sentinel-2 imagery. -
More efficient computation during inference by employing a 64-pixel execution point for the MEOX encoder, which is noted as
the efficient operating point,
reducing computational load compared to larger grids while maintaining competitive accuracy. -
Better diagnostic capabilities through routing diagnostics that distinguish
expert participation, spatial dependence, modality association, and functional contribution,
providing an inspectable understanding of how the model utilizes its sparse mixture-of-experts layers. -
Improved sensor adaptation by utilizing frozen transfer on six GEO-Bench tasks and employing BigEarthNet finetuning to reach a 72.95% micro-average precision, demonstrating strong task transfer capability with a compact model.
Abstract
Multimodal Earth Observation models often fuse radar and optical inputs early, limiting sensor-conditioned processing, or maintain separate sensor streams through the encoder, increasing computational cost. We present MEOX (Multimodal Earth Observation with eXperts), which delays fusion until after a shared mixture-of-experts block that processes Sentinel-2 and ascending and descending Sentinel-1 streams independently. Co-located patches are then fused with validity-aware weights into a single sequence processed by fourteen shared blocks, reducing the sequence length from 3N + 5 to N + 5 tokens for N patches. We pretrain the 2.939M-parameter encoder on 1.228M MMEarth64 samples using masked radar-optical reconstruction. Under the reported frozen-transfer protocols, MEOX reaches 90.56% average class accuracy on EuroSAT and 64.42% mean intersection over union (mIoU) on cashew segmentation, 2.26 and 5.02 points above the strongest reported cross-sensor mixture-of-experts (CSMoE) variants. Full-encoder adaptation increases BigEarthNet micro-average precision from 57.22% to 72.95%. In a controlled same-corpus test using synthetic random patch removal, masking 75% of Sentinel-2 patches reduces WorldCover mIoU from 30.45% to 26.53% when both Sentinel-1 streams remain available; with Sentinel-2 alone, mIoU falls to 13.81%. For identical zeroed optical patches, declaring them missing through the validity mechanism improves mIoU from 19.42% to 26.53%. These results characterize a compact route-then-fuse encoder that retains sensor-conditioned processing and uses complementary radar observations when optical inputs are incomplete.
Sources
- Neural Plasticity-Inspired Multimodal Foundation Model for Earth Observation
- The Rise of Sparse Mixture-of-Experts: A Survey from Algorithmic Foundations to Decentralized Architectures and Vertical Domain Applications
- Lightweight Metadata-Aware Mixture-of-Experts Masked Autoencoder for Earth Observation
- Lightweight, Pre-trained Transformers for Remote Sensing Timeseries
- Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer
- CSMoE: An Efficient Remote Sensing Foundation Model with Soft Mixture-of-Experts
- LoRA: Low-Rank Adaptation of Large Language Models
- GLU Variants Improve Transformer
Related papers
- TEA: Text Encoder Alignment for Robust Concept Erasure in Text-to-Image Models
- Do Egocentric Video-Language Models Capture Both Hand- and Object-Centric Cues?
- Aura: Consistent Multi-Subject Video Generation via VLM-Grounded Semantic Alignment
- PVRF: All-in-one Adverse Weather Removal via Prior-modulated and Velocity-constrained Rectified Flow
- DLWM: Diverse Latent World Models for Efficient Multimodal Reasoning
- Out of Sight, Still in Mind: Token Compression for Omni-LLMs