MEOX: Modality-Conditioned Routing Before Fusion in a Compact Earth Observation Encoder

summary

The gist

MEOX presents a compact multimodal masked autoencoder designed for Earth Observation that preserves modality-dependent processing before a learned patch-wise fusion The model achieves strong task…

In short

MEOX is a compact multimodal autoencoder for Earth Observation that processes sensor data separately before fusing them. It uses sensor-specific adapters and a shared sparse-expert block to handle different data types efficiently. The model achieves strong task transfer using a small number of parameters by carefully managing how missing data affects the reconstruction process.

Key concepts

Multimodal Masked Autoencoder
This is an AI model designed to understand multiple types of input, like different satellite images, simultaneously. It learns to reconstruct missing parts of the input data by looking at all available pieces across different modalities (sensors) at once. It's a way for the model to learn how different data types relate to each other.
Sensor-Specific Adapters
These are small, specialized modules added within the main encoder that are tailored to handle the unique characteristics of each sensor. This allows the model to process data from one sensor in a way that is optimized for its specific format and noise, ensuring modality-dependent processing before any merging happens.
Patch-wise Fusion Score
This mechanism calculates a score for how important each available sensor's information is at every single location (patch) on the image. This score determines the weight given to each sensor's contribution during the final fusion step, ensuring that only relevant data influences the combined output.
Sparse-Expert Block
This is a shared component within the model that uses a sparse expert system. Instead of using every single parameter for every calculation, it selectively activates only a few specialized parts when needed. This keeps the total number of parameters small while still allowing the model to perform complex, modality-aware computations.

Terminology used across episodes

This episode discusses

The paper

MEOX: Modality-Conditioned Routing Before Fusion in a Compact Earth Observation Encoder · Read on arXiv

European Centre for Medium-Range Weather Forecasts

Multimodal Earth Observation models often fuse radar and optical inputs early, limiting sensor-conditioned processing, or maintain separate sensor streams through the encoder, increasing computational cost. We present MEOX (Multimodal Earth Observation with eXperts), which delays fusion until after a shared mixture-of-experts block that processes Sentinel-2 and ascending and descending Sentinel-1 streams independently. Co-located patches are then fused with validity-aware weights into a single sequence processed by fourteen shared blocks, reducing the sequence length from 3N + 5 to N + 5 tokens for N patches. We pretrain the 2.939M-parameter encoder on 1.228M MMEarth64 samples using masked radar-optical reconstruction. Under the reported frozen-transfer protocols, MEOX reaches 90.56% average class accuracy on EuroSAT and 64.42% mean intersection over union (mIoU) on cashew segmentation, 2.26 and 5.02 points above the strongest reported cross-sensor mixture-of-experts (CSMoE) variants. Full-encoder adaptation increases BigEarthNet micro-average precision from 57.22% to 72.95%. In a controlled same-corpus test using synthetic random patch removal, masking 75% of Sentinel-2 patches reduces WorldCover mIoU from 30.45% to 26.53% when both Sentinel-1 streams remain available; with Sentinel-2 alone, mIoU falls to 13.81%. For identical zeroed optical patches, declaring them missing through the validity mechanism improves mIoU from 19.42% to 26.53%. These results characterize a compact route-then-fuse encoder that retains sensor-conditioned processing and uses complementary radar observations when optical inputs are incomplete.

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "MEOX: Modality-Conditioned Routing Before Fusion in a Compact Earth Observation Encoder".

Jane: Here's the gist of the paper. MEOX presents a compact multimodal masked autoencoder designed for Earth Observation that preserves modality-dependent processing before a learned patch-wise fusion The model achieves strong task transfer using a compact parameter budget.

Tom: Model Architecture and Components MEOX is a multimodal masked autoencoder with a 2.939 million-parameter encoder and 3.115 million parameters in total.

Jane: The architecture incorporates sensor-specific adapters, explicit validity signals, and a shared sparse-expert block to maintain modality-dependent processing before patch-wise fusion.

Tom: Four metadata tokens accompany a single spatial sequence through fourteen further encoder blocks.

Paper summary: Tom: So, looking at the MEOX paper and its title, Modality-Conditioned Routing Before Fusion in a Compact Earth Observation Encoder—it’s essentially about designing a smarter way for AI to look at messy, multi-sensor satellite data.

Jane: It tackles the issue of combining different sensor types by making sure each one gets its own focused processing path before they are merged. The authors show that this approach works well on things like six GEO-Bench tasks when tested at both sixty-four and two hundred twenty-four pixels, achieving results that beat some other models <ref:2609.05351#pg1,at both 64 and 224 pixels>.

Lu: What this means in simple terms is that instead of having one giant neural network trying to figure out how to combine all the satellite bands at once, you have a system that learns which sensor provides the best local context for every single pixel and fuses them based on those learned weights.

Meng: From a deployment standpoint, the compact parameter budget and the explicit handling of missing data suggest this is something we could actually deploy in operational settings without needing an enormous computational footprint just to get decent reconstruction results.

Lalam: I'm confident in what the literature points to; this approach provides a structured way for heterogeneous sensors to communicate their relevance before they ever get mixed together into the final output.

Tom: So, MEOX offers a compact architecture that uses sensor-specific adapters and explicit validity signals to maintain modality-dependent processing through an entire encoder, followed by a learned patch-wise fusion step. It’s about efficient routing before the heavy lifting of fusion.

Conclusion: Tom: So we've been looking at MEOX, which stands for Modality-Conditioned Routing Before Fusion in a Compact Earth Observation Encoder by the authors of this paper.

Jane: It basically shows how you can process different satellite images—like infrared and visible light—separately before you try to mix them together.

Lu: The core idea is using sensor-specific adapters and explicit validity signals to keep those modalities distinct throughout the encoding process.

Meng: That sounds like a lot of complexity if we're trying to ship something small and fast for production systems, you know?

Lalam: But they use a shared sparse expert block that keeps the parameter count down while still allowing that specialized processing before the fusion step.

Tom: So, they’ve managed to keep a pretty tight parameter budget while maintaining this modality-dependent flow.

Jane: And when we look at their results, they show strong task transfer using those compact parameters, which means it learns well across different tasks on the data.

Lu: They also use four extra metadata tokens alongside the spatial sequence—latitude, longitude, month, and climate context—to give the model more information about where and when it's looking.

Meng: That metadata stuff is interesting because it helps handle missing data differently; if you have a few pieces of that info missing, the contribution of that missing part to the final result just goes up.

Lalam: So, instead of filling in zeros everywhere, they’re letting the model know how much it needs to fill in based on what's actually available.

Tom: And they test it on six different GEO-Bench tasks at both sixty-four and two hundred twenty-four pixels, and those results beat some other models reported for this specific setup.

Jane: It suggests that this routing mechanism before fusion is a solid way to handle heterogeneous data in Earth observation without needing massive computational resources for every sensor combination.

Lu: The implication is that we can build more efficient encoders where the model intelligently weighs which sensor signal matters most at each exact spot on the image.

Meng: I wonder if those specific adapters are easy to fine-tune when we get new satellite data sources, or if they need a whole re-training process every time.

Lalam: They constrain parameter growth with these shared expert projections, which helps keep the model from just growing infinitely with every new sensor you throw at it.

Tom: It's about keeping the processing localized and focused on the specific sensor input before the final combination happens.

Jane: It’s a neat way to handle messy, multi-sensor data by making sure each piece gets its own thoughtful consideration before they merge into one final picture.

Lu: So, moving forward, we should look closely at how they handle those validity signals and see if that's a robust way to manage real-world sensor noise.

More episodes

← Home