MEOX: Modality-Conditioned Routing Before Fusion in a Compact Earth Observation Encoder

arXiv:2609.05351 · cs.CV · Submitted 2026-09-04 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "MEOX: Modality-Conditioned Routing Before Fusion in a Compact Earth Observation Encoder".

Jane: Here's the gist of the paper. MEOX presents a compact multimodal masked autoencoder designed for Earth Observation that preserves modality-dependent processing before a learned patch-wise fusion The model achieves strong task transfer using a compact parameter budget.

Tom: Model Architecture and Components MEOX is a multimodal masked autoencoder with a 2.939 million-parameter encoder and 3.115 million parameters in total.

Jane: The architecture incorporates sensor-specific adapters, explicit validity signals, and a shared sparse-expert block to maintain modality-dependent processing before patch-wise fusion.

Tom: Four metadata tokens accompany a single spatial sequence through fourteen further encoder blocks.

Paper summary: Tom: So, looking at the MEOX paper and its title, Modality-Conditioned Routing Before Fusion in a Compact Earth Observation Encoder—it’s essentially about designing a smarter way for AI to look at messy, multi-sensor satellite data.

Jane: It tackles the issue of combining different sensor types by making sure each one gets its own focused processing path before they are merged. The authors show that this approach works well on things like six GEO-Bench tasks when tested at both sixty-four and two hundred twenty-four pixels, achieving results that beat some other models <ref:2609.05351#pg1,at both 64 and 224 pixels>.

Lu: What this means in simple terms is that instead of having one giant neural network trying to figure out how to combine all the satellite bands at once, you have a system that learns which sensor provides the best local context for every single pixel and fuses them based on those learned weights.

Meng: From a deployment standpoint, the compact parameter budget and the explicit handling of missing data suggest this is something we could actually deploy in operational settings without needing an enormous computational footprint just to get decent reconstruction results.

Lalam: I'm confident in what the literature points to; this approach provides a structured way for heterogeneous sensors to communicate their relevance before they ever get mixed together into the final output.

Tom: So, MEOX offers a compact architecture that uses sensor-specific adapters and explicit validity signals to maintain modality-dependent processing through an entire encoder, followed by a learned patch-wise fusion step. It’s about efficient routing before the heavy lifting of fusion.

Conclusion: Tom: So we've been looking at MEOX, which stands for Modality-Conditioned Routing Before Fusion in a Compact Earth Observation Encoder by the authors of this paper.

Jane: It basically shows how you can process different satellite images—like infrared and visible light—separately before you try to mix them together.

Lu: The core idea is using sensor-specific adapters and explicit validity signals to keep those modalities distinct throughout the encoding process.

Meng: That sounds like a lot of complexity if we're trying to ship something small and fast for production systems, you know?

Lalam: But they use a shared sparse expert block that keeps the parameter count down while still allowing that specialized processing before the fusion step.

Tom: So, they’ve managed to keep a pretty tight parameter budget while maintaining this modality-dependent flow.

Jane: And when we look at their results, they show strong task transfer using those compact parameters, which means it learns well across different tasks on the data.

Lu: They also use four extra metadata tokens alongside the spatial sequence—latitude, longitude, month, and climate context—to give the model more information about where and when it's looking.

Meng: That metadata stuff is interesting because it helps handle missing data differently; if you have a few pieces of that info missing, the contribution of that missing part to the final result just goes up.

Lalam: So, instead of filling in zeros everywhere, they’re letting the model know how much it needs to fill in based on what's actually available.

Tom: And they test it on six different GEO-Bench tasks at both sixty-four and two hundred twenty-four pixels, and those results beat some other models reported for this specific setup.

Jane: It suggests that this routing mechanism before fusion is a solid way to handle heterogeneous data in Earth observation without needing massive computational resources for every sensor combination.

Lu: The implication is that we can build more efficient encoders where the model intelligently weighs which sensor signal matters most at each exact spot on the image.

Meng: I wonder if those specific adapters are easy to fine-tune when we get new satellite data sources, or if they need a whole re-training process every time.

Lalam: They constrain parameter growth with these shared expert projections, which helps keep the model from just growing infinitely with every new sensor you throw at it.

Tom: It's about keeping the processing localized and focused on the specific sensor input before the final combination happens.

Jane: It’s a neat way to handle messy, multi-sensor data by making sure each piece gets its own thoughtful consideration before they merge into one final picture.

Lu: So, moving forward, we should look closely at how they handle those validity signals and see if that's a robust way to manage real-world sensor noise.

European Centre for Medium-Range Weather Forecasts

cs.CV

Submitted: 2026-09-04

Updated: 2026-11-03

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 90/100

The gist: MEOX presents a compact multimodal masked autoencoder designed for Earth Observation that preserves modality-dependent processing before a learned patch-wise fusion The model achieves strong task…

Key concepts

Multimodal Masked Autoencoder
This is an AI model designed to understand multiple types of input, like different satellite images, simultaneously. It learns to reconstruct missing parts of the input data by looking at all available pieces across different modalities (sensors) at once. It's a way for the model to learn how different data types relate to each other.
Sensor-Specific Adapters
These are small, specialized modules added within the main encoder that are tailored to handle the unique characteristics of each sensor. This allows the model to process data from one sensor in a way that is optimized for its specific format and noise, ensuring modality-dependent processing before any merging happens.
Patch-wise Fusion Score
This mechanism calculates a score for how important each available sensor's information is at every single location (patch) on the image. This score determines the weight given to each sensor's contribution during the final fusion step, ensuring that only relevant data influences the combined output.
Sparse-Expert Block
This is a shared component within the model that uses a sparse expert system. Instead of using every single parameter for every calculation, it selectively activates only a few specialized parts when needed. This keeps the total number of parameters small while still allowing the model to perform complex, modality-aware computations.

Terminology

Summary

MEOX presents a compact multimodal masked autoencoder designed for Earth Observation that preserves modality-dependent processing before a learned patch-wise fusion The model achieves strong task transfer using a compact parameter budget. p. 0

Where the paper says it

These results demonstrate sensor-flexible representation learning and strong task transfer using a compact parameter budget. Index Terms—Earth observation, multimodal learning, masked autoencoders, mixture of experts, remote sensing, compact representation learning.

Model Architecture and Components

MEOX is a multimodal masked autoencoder with a 2.939 million-parameter encoder and 3. p. 0

Where the paper says it

We present MEOX (Multimodal Earth Observation with eXperts), a multimodal masked autoencoder with a 2. 939 million-parameter encoder and 3.

115 million parameters in total. p. 0
Where the paper says it

939 million-parameter encoder and 3. 115 million parameters in total.

The architecture incorporates sensor-specific adapters, explicit validity signals, and a shared sparse-expert block to maintain modality-dependent processing before patch-wise fusion. p. 0
Where the paper says it

115 million parameters in total. Sensor-specific adapters, explicit validity signals, and a shared sparse-expert block preserve modality-dependent processing before a learned patch-wise fusion.

Four metadata tokens accompany a single spatial sequence through fourteen further encoder blocks. p. 0
Where the paper says it

Sensor-specific adapters, explicit validity signals, and a shared sparse-expert block preserve modality-dependent processing before a learned patch-wise fusion. Four metadata tokens then accompany a single spatial sequence through fourteen further encoder blocks.

Shared expert projections with private low-rank residuals constrain parameter growth, while rotary attention supports downstream spatial grids different from pretraining. p. 0
Where the paper says it

Four metadata tokens then accompany a single spatial sequence through fourteen further encoder blocks. Shared expert projections with private low-rank residuals constrain parameter growth, while rotary attention supports downstream spatial grids different from pretraining.

Data Representation and Missingness Handling

Missing elements are numerically zero-filled only after validity is retained, meaning a missing value differs from a valid observation equal to its normalization mean. Each sensor image is divided into nonoverlapping P × P patches, resulting in N = HW/P2 spatial tokens. p. 1

Where the paper says it

Each sensor image is divided into nonoverlapping P × P patches. An image of size H × W therefore produces N = HW/P2 spatial tokens.

Metadata is represented by four additional tokens, one each for latitude, longitude, month, and ERA5 climate context. p. 1
Where the paper says it

With H = W = 64 and P = 4, each sensor produces N = 16 × 16 = 256 tokens of width d = 144. Metadata is represented by four additional tokens, one each for latitude, longitude, month, and ERA5 climate context.

The missing-state contribution is zero when all components are present and increases as more components become unavailable. p. 1
Where the paper says it

The first term projects the available values, ϵ type j identifies the metadata group, and ϵ miss j represents missingness. The missing-state contribution is zero when all components are present and increases as more components become unavailable.

Fusion Mechanism and Expert Routing

Before fusion, MEOX processes each sensor independently using the same transformer-MoE block, producing a representation zm = Fθ(tm). At each spatial position, the model computes one fusion score for every available sensor using Equation (3). p. 2

Where the paper says it

Attention and expert routing therefore operate on sensor-specific observations before cross-sensor fusion. At each spatial position, the model computes one fusion score for every available sensor: am,i = w ⊤ f LN(zm,i) + cm + logmax(qm,i, 10−6 ) , (3) αm,i = exp(am,i) P m′∈A exp(am′ ,i) , zi = X m∈A αm,izm,i.

The quantity qm,i ∈ [0, 1] is the fraction of valid elements in patch i of sensor m. p. 2
Where the paper says it

(4) Here, am,i is the unnormalized fusion score, wf ∈ R d is a learned vector that scores the normalized patch representation, and cm is a learned sensor-specific bias. The quantity qm,i ∈ [0, 1] is the fraction of valid elements in patch i of sensor m.

The softmax produces nonnegative fusion weights αm,i that sum to one across the available sensors. p. 2
Where the paper says it

The lower bound 10−6 prevents numerical evaluation of log(0). The softmax produces nonnegative fusion weights αm,i that sum to one across the available sensors.

Training Objectives and Evaluation

The pretraining objective balances modalities by valid target elements and reconstructs deliberately omitted sensors. The reconstruction loss for sensor m is calculated only on spatially masked and valid target values, defined by Equation (9). p. 3

Where the paper says it

The reconstruction loss is calculated only on spatially masked and valid target values. Let m identify a sensor, n a sample in the batch, i a spatial patch, and k an element within that patch, including its pixel and channel position.

An auxiliary term encourages balanced expert routing, defined by Equation (14). p. 4
Where the paper says it

Equal sensor averaging prevents this channel-count difference from dominating training. The final objective also includes an auxiliary term that encourages balanced expert routing.

Frozen transfer is evaluated on six GEO-Bench tasks at both 64 and 224 pixels, achieving results exceeding corresponding reported CSMoE outcomes. p. 0
Where the paper says it

228 million MMEarth64 samples using modality-balanced masked reconstruction and structured sensor dropout. Frozen transfer is evaluated on six GEO-Bench tasks at both 64 and 224 pixels.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Improved representation learning for Earth Observation by using MEOX, which is a multimodal masked autoencoder with 2.939 million parameters, enabling sensor-flexible representation learning and strong task transfer using a compact parameter budget.

  2. Enhanced handling of missing data through explicit validity signals and metadata tokens, allowing the model to distinguish unavailable values from valid normalized zeros when processing Sentinel-1 or Sentinel-2 imagery.

  3. More efficient computation during inference by employing a 64-pixel execution point for the MEOX encoder, which is noted as the efficient operating point, reducing computational load compared to larger grids while maintaining competitive accuracy.

  4. Better diagnostic capabilities through routing diagnostics that distinguish expert participation, spatial dependence, modality association, and functional contribution, providing an inspectable understanding of how the model utilizes its sparse mixture-of-experts layers.

  5. Improved sensor adaptation by utilizing frozen transfer on six GEO-Bench tasks and employing BigEarthNet finetuning to reach a 72.95% micro-average precision, demonstrating strong task transfer capability with a compact model.

Abstract

Multimodal Earth Observation models often fuse radar and optical inputs early, limiting sensor-conditioned processing, or maintain separate sensor streams through the encoder, increasing computational cost. We present MEOX (Multimodal Earth Observation with eXperts), which delays fusion until after a shared mixture-of-experts block that processes Sentinel-2 and ascending and descending Sentinel-1 streams independently. Co-located patches are then fused with validity-aware weights into a single sequence processed by fourteen shared blocks, reducing the sequence length from 3N + 5 to N + 5 tokens for N patches. We pretrain the 2.939M-parameter encoder on 1.228M MMEarth64 samples using masked radar-optical reconstruction. Under the reported frozen-transfer protocols, MEOX reaches 90.56% average class accuracy on EuroSAT and 64.42% mean intersection over union (mIoU) on cashew segmentation, 2.26 and 5.02 points above the strongest reported cross-sensor mixture-of-experts (CSMoE) variants. Full-encoder adaptation increases BigEarthNet micro-average precision from 57.22% to 72.95%. In a controlled same-corpus test using synthetic random patch removal, masking 75% of Sentinel-2 patches reduces WorldCover mIoU from 30.45% to 26.53% when both Sentinel-1 streams remain available; with Sentinel-2 alone, mIoU falls to 13.81%. For identical zeroed optical patches, declaring them missing through the validity mechanism improves mIoU from 19.42% to 26.53%. These results characterize a compact route-then-fuse encoder that retains sensor-conditioned processing and uses complementary radar observations when optical inputs are incomplete.

Sources

Related papers