MMEarth-Bench: Global Model Adaptation via Multimodal Test-Time Training

arXiv:2602.06285 · cs.CV · Submitted 2026-02-06 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "MMEarth-Bench: Global Model Adaptation via Multimodal Test-Time Training".

Jane: Here's the gist of the paper. Recent research in geospatial machine learning shows that models pretrained with self-supervised learning on Earth observation data can perform well on downstream tasks even when labeled data is limited.

Tom: However, existing benchmark datasets often suffer from having few data modalities and poor global representation, which hinders the evaluation of multimodal pretrained models at a global scale.

Jane: To address this gap, the authors introduce MMEarth-Bench, a new multimodal benchmark dataset comprising five environmental monitoring tasks across 12 aligned Earth observation (EO) modalities.

Tom: This dataset provides globally distributed data and includes both random and geographic test splits for robust benchmarking.

Paper summary: Tom: So wrapping up this paper, MMEarth-Bench gives us a complete set of environmental monitoring tasks and data types to really test how well pretrained models perform when they have limited labeled data or need to adapt to new geographic areas.

Jane: And the authors are demonstrating that while self-supervised multimodal pretraining helps in those low-shot regimes and under geographic shifts, the TTTMMR approach gives us a way to adapt those models efficiently at test time by leveraging all available modalities as reconstruction tasks.

Lu: The study suggests that TTTMMR-Geo is the best method for most models, although they also pointed out that it isn't perfect, and some specific models still show limitations when using this adaptation technique.

Meng: From an engineering view, this means we have a standardized way to improve model adaptation without having to retrain the whole thing from scratch every time you move to a new region.

Lalam: It confirms that leveraging all modalities at test time as reconstruction tasks is a solid strategy for boosting performance in these complex geospatial scenarios.

Conclusion: Tom: So we've seen how they built this whole new testing ground called MMEarth-Bench, and now we need to talk about what that title means for us.

Jane: It’s titled "MMEarth-Bench: Global Model Adaptation via Multimodal Test-Time Training." That sounds like a lot of jargon for something that’s actually about how models work when they have to change their minds.

Tom: Exactly. The core idea here is taking these models that are already trained on tons of data and figuring out how to make them perform well, even when they move to a completely new geographic area or see a different type of data they weren't trained on.

Lu: What the authors are really pushing here is that self-supervised multimodal pretraining helps with those low-shot scenarios and shifting geography, but it's not perfect across all tasks.

Jane: They found that while this pretraining is good, geographic generalization still struggles on some of the tasks they tested. It stops working reliably for about four out of five monitoring goals they looked at.

Tom: And the big takeaway from their work is introducing TestTime Training with MultiModal Reconstruction, or TTT-MMR, as a way to fix that adaptation problem at test time without retraining the whole thing.

Jane: So you’re telling us they use all the different data types available—all twelve of them—as extra training tasks during the test phase to help the model adjust quickly.

Lu: Right. They normalize those reconstruction losses across all modalities so every single one contributes fairly to that adaptation signal, no matter if it was a big or small dataset originally.

Tom: I’m seeing results where TTT-MMR and a version called Geographic Batching actually beat training the model from scratch in many cases, which is huge for saving time on new deployments.

Jane: That means for someone deploying an AI system into a remote part of the world, they don't have to wait months to collect enough labeled data just to get it working.

Tom: It shows that we can leverage existing knowledge across different sensors—RGB, multispectral, whatever—to make models more robust when the real-world conditions change.

Lu: The authors highlight that this method works best for most models except for a couple of specific ones, which tells you exactly where the current limitations are in the technology itself.

Jane: It’s a solid step forward, showing that we can bridge the gap between training models on clean data and making them useful in messy, real-world environments.

Tom: And this opens up a lot of discussion about how we should be designing those next generation benchmarks to really test these new adaptation techniques.

Harvard University · University of Copenhagen

cs.CV

Submitted: 2026-02-06

Updated: 2026-11-11

Comments: Published at ECCV 2026, project page: https://lgordon99.github.io/mmearth-bench/

DOI: 10.1007/978-3-032-37252-9_19

Project page: https://lgordon99.github.io/mmearth-bench-app

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 99/100

The gist: Recent research in geospatial machine learning shows that models pretrained with self-supervised learning on Earth observation data can perform well on downstream tasks even when labeled data is…

Key concepts

MMEarth-Bench
This is a new multimodal benchmark dataset comprising five environmental monitoring tasks across 12 aligned Earth observation (EO) modalities. It provides globally distributed data and includes both random and geographic test splits to robustly evaluate how well models generalize across different types of environmental data.
Multimodal Pretraining
This involves training models using self-supervised learning on diverse Earth observation data, such as RGB or multispectral images. The goal is to teach the model general features from many different sources so it can perform well on new, related tasks even when labeled training data is scarce.
TestTime Training with MultiModal Reconstruction (TTT-MMR)
This method adapts a pretrained model at test time by treating all available modalities as auxiliary self-supervised tasks. It uses modality reconstruction losses as the adaptation signal, normalizing gradients across modalities to ensure every input type contributes equally to the model's learning process.
Geographic Batching (TTT-MMR-Geo)
This variant of TTTMMR introduces a trade-off between regularization and specialization during test-time adaptation. It is particularly effective at improving performance on rare events or long-tail distributions, making it robust to sample rarity in regression tasks.

Terminology

Summary

Recent research in geospatial machine learning shows that models pretrained with self-supervised learning on Earth observation data can perform well on downstream tasks even when labeled data is limited. p. 0

Where the paper says it

dk Abstract. Recent research in geospatial machine learning demonstrates that models pretrained with self-supervised learning on Earth observation data can perform well on downstream tasks with limited labeled data.

However, existing benchmark datasets often suffer from having few data modalities and poor global representation, which hinders the evaluation of multimodal pretrained models at a global scale. p. 0
Where the paper says it

Recent research in geospatial machine learning demonstrates that models pretrained with self-supervised learning on Earth observation data can perform well on downstream tasks with limited labeled data. However, most benchmark datasets have few data modalities and poor global representation, limiting the ability to evaluate multimodal pretrained models at global scales.

To address this gap, the authors introduce MMEarth-Bench, a new multimodal benchmark dataset comprising five environmental monitoring tasks across 12 aligned Earth observation (EO) modalities. p. 2
Where the paper says it

MMEarth-Bench dataset: We introduce five multimodal environmental monitoring tasks for benchmarking pretrained models. Each task consists of 12 aligned modalities and provides globally distributed data with both random and geographic test splits.

This dataset provides globally distributed data and includes both random and geographic test splits for robust benchmarking. p. 2
Where the paper says it

MMEarth-Bench dataset: We introduce five multimodal environmental monitoring tasks for benchmarking pretrained models. Each task consists of 12 aligned modalities and provides globally distributed data with both random and geographic test splits.

The study benchmarks eight recent pretrained models, including those with RGB, multispectral, and multimodal pretraining strategies. p. 2

Where the paper says it

Benchmarking: We benchmark 8 recent pretrained models covering a range of architectures and including both uni- and multimodal models. We find that all models still struggle to generalize geographically and that training a multimodal model from scratch is competitive with global pretraining.

A key finding is that while multimodal pretraining generally enhances model robustness in low-shot regimes and under geographic shifts, the geographic generalization abilities remain poor across four out of five tasks. p. 0
Where the paper says it

In order to fill this gap, we introduce MMEarth-Bench, a collection of five new environmental tasks with 12 modalities, globally distributed data, and both random and geographic test splits. We benchmark a diverse set of pretrained models and find that while (multimodal) pretraining tends to improve model robustness in limited data settings, geographic generalization abilities remain poor.

Furthermore, the authors find that a simple multimodal model trained from scratch can be competitive given sufficient labeled data. p. 0
Where the paper says it

We benchmark a diverse set of pretrained models and find that while (multimodal) pretraining tends to improve model robustness in limited data settings, geographic generalization abilities remain poor. Moreover, a simple randomly initialized multimodal model is competitive given enough labeled data.

To overcome the limitation where models can only utilize modalities they were pretrained on, they propose TestTime Training with MultiModal Reconstruction (TTT-MMR). p. 2

Where the paper says it

Complementary to using multimodal data for self-supervised pretraining, we propose using the modalities as reconstruction tasks in the regime of test-time adaptation, which uses unlabeled test data to adapt the model at test time. We formulate our proposed method for TestTime Training with MultiModal Reconstruction (TTT-MMR), along with a second variant using geographic batching (TTT-MMR-Geo), which leads to a good trade-off between specialization and regularization, a crucial aspect in TTT [25, 39].

This method repurposes all available modalities at test time as auxiliary self-supervised tasks, using modality reconstruction losses as a test-time adaptation signal. p. 2
Where the paper says it

To address this inefficiency, we introduce a method for using all available modalities at test time, whether or not they are compatible with the pretrained model. Complementary to using multimodal data for self-supervised pretraining, we propose using the modalities as reconstruction tasks in the regime of test-time adaptation, which uses unlabeled test data to adapt the model at test time.

The TTTMMR approach normalizes the gradients of these task modality reconstruction losses by modality and then averages them across modalities to ensure each modality contributes equally to the adaptation signal, irrespective of its original scale. p. 7
Where the paper says it

Then at test time the reconstruction losses serve as an adaptation signal for the encoder (see Fig. TTT-MMR normalizes the gradients of the task modality reconstruction losses by modality and then averages them across modalities (excluding missing modalities) so that each modality contributes equally to the adaptation signal, regardless of its original scale.

A variant, Geographic Batching (TTT-MMR-Geo), is proposed to further improve performance by introducing a trade-off between regularization and specialization during test-time adaptation. p. 2

Where the paper says it

Complementary to using multimodal data for self-supervised pretraining, we propose using the modalities as reconstruction tasks in the regime of test-time adaptation, which uses unlabeled test data to adapt the model at test time. We formulate our proposed method for TestTime Training with MultiModal Reconstruction (TTT-MMR), along with a second variant using geographic batching (TTT-MMR-Geo), which leads to a good trade-off between specialization and regularization, a crucial aspect in TTT [25, 39].

Experiments consistently show that both TTTMMR and TTTMMR-Geo outperform joint training (JT) across all tasks and splits. p. 0
Where the paper says it

To solve this problem, we propose using all the modalities available at test time as auxiliary tasks for test-time adaptation. Our model-agnostic method for test-time training with multimodal reconstruction (TTTMMR) can improve performance across all models and tasks on both test splits.

Specifically, geographic batching is shown to be particularly beneficial for improving performance on the long-tail of target distributions, proving robust to sample rarity in regression tasks. p. 0
Where the paper says it

Our model-agnostic method for test-time training with multimodal reconstruction (TTTMMR) can improve performance across all models and tasks on both test splits. Furthermore, geographic batching leads to a good trade-off between regularization and specialization during TTT, which is especially beneficial for long-tail distributions.

The results indicate that while all benchmarked models benefit from the proposed methods, TTTMMR-Geo is the best method for most models except SatlasNet, Galileo, and ConvNeXtV2A-MM. p. 9

Where the paper says it

Table 4: Pretrained models benchmarked. Method Architecture Pretraining dataset Input modalities Scale-MAE [50] ViT-L FMoW-RGB [12] RGB DINOv3 Web [55] ViT-L/16 distilled LVD-1689M [55] RGB DINOv3 Sat [55] ViT-L/16 distilled SAT-493M [55] RGB SatlasNet [5] Swin-v2-B SatlasPretrain [5] S2 MPMAE [42] ConvNeXt V2-A MMEarth [42] S2 TerraMind [29] TerraMindv1-B TerraMesh [8] S2, S1, DEM, RGB Copernicus-FM [62] ViT-B Copernicus-Pretrain [62] S2, S1, DEM, Geolocation, Date Galileo [60] ViT-B Galileo dataset [60] S2, S1, NDVI, Temperature, Precipitation, DEM, Dynamic World, Geolocation, Month (ConvNeXtV2A-MM) with all modalities as input.

The gains are not dominated by a single modality; for instance, canopy height proves particularly useful for the biomass task. p. 7
Where the paper says it

As we do not use test labels, we treat these modalities as proxies for our task of interest. While we know that canopy height is correlated with biomass [17, 32], for example, it is generally nontrivial to preselect the best proxies for a given task.

The analysis also reveals that long-tail performance improves especially with TTT-MMR-Geo. p. 0
Where the paper says it

Our model-agnostic method for test-time training with multimodal reconstruction (TTTMMR) can improve performance across all models and tasks on both test splits. Furthermore, geographic batching leads to a good trade-off between regularization and specialization during TTT, which is especially beneficial for long-tail distributions.

For regression tasks, TTT leads to the greatest performance gains on the random split for species rather than the geographic split of soil organic carbon, where R2 values are mostly negative before and after adaptation. p. 6
Where the paper says it

MMEarth-Bench 7 Table 2: MMEarth-Bench tasks. Task # Tiles Unit Scale Type License Biomass 18,393 Mg/ha Pixel-level Regression CC BY Soil Nitrogen 5,643 g/kg Tile-level Regression CC BY-NC Soil Organic Carbon 7,982 g/kg Tile-level Regression CC BY-NC Soil pH 8,508 Unitless Tile-level Regression CC BY-NC Species 36,410 Occurrence Tile-level Multi-label classification Terms of Use Table 3: MMEarth-Bench modalities.

In conclusion, MMEarth-Bench serves as a comprehensive multimodal benchmark dataset. p. 1

Where the paper says it

While existing EO benchmarks [9, 22, 31,38, 62] have helped to develop and evaluate pretrained geospatial models, they limit the evaluation of progress towards overcoming these three grand challenges, as most of their downstream tasks are i) regional rather than global, ii) not evaluated on geographic splits with an entire held-out region, and iii) only provide few modalities per task. In this work we introduce MMEarth-Bench, a multimodal EO benchmark dataset containing five global tasks designed to evaluate pretrained geospatial models in these three challenging yet crucial settings.

The authors demonstrate that self-supervised multimodal pretraining helps in low-shot regimes and under geographic shifts, but TTTMMR provides an efficient, model-agnostic method for improving adaptation performance at test time by leveraging all modalities as reconstruction tasks. p. 0
Where the paper says it

To solve this problem, we propose using all the modalities available at test time as auxiliary tasks for test-time adaptation. Our model-agnostic method for test-time training with multimodal reconstruction (TTTMMR) can improve performance across all models and tasks on both test splits.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Multimodal adaptation signal using all available modalities at test time (TTT-MMR) can be used to improve performance across all models and tasks on both test splits, as it improves performance across all models and tasks on both test splits.

  2. Geographic batching in the TTT process (TTT-MMR-Geo) leads to a good trade-off between regularization and specialization during adaptation, which is especially beneficial for long-tail distributions.

  3. The proposed method allows for model adaptation when labels are limited and geographically biased by repurposing multimodal pretext tasks for testtime adaptation, using modality reconstruction losses as a test-time adaptation signal.

  4. The system can be designed to use all modalities available at test time as auxiliary tasks for inference, addressing the inefficiency where models can currently only make use of the modalities on which they were pretrained.

  5. Geographic batching provides unique robustness to rare samples for regression tasks, as it allows more specialization to the geographic domain defined by the batch, which is crucial when dealing with long-tail distributions.

Abstract

Recent research in geospatial machine learning demonstrates that models pretrained with self-supervised learning on Earth observation data can perform well on downstream tasks with limited labeled data. However, most benchmark datasets have few data modalities and poor global representation, limiting the ability to evaluate multimodal pretrained models at global scales. In order to fill this gap, we introduce MMEarth-Bench, a collection of five new environmental tasks with 12 modalities, globally distributed data, and both random and geographic test splits. We benchmark a diverse set of pretrained models and find that while (multimodal) pretraining tends to improve model robustness in limited data settings, geographic generalization abilities remain poor. Moreover, a simple randomly initialized multimodal model is competitive given enough labeled data. Although data is abundant, models can currently only make use of the modalities on which they were pretrained. To solve this problem, we propose using all the modalities available at test time as auxiliary tasks for test-time adaptation. Our model-agnostic method for test-time training with multimodal reconstruction (TTT-MMR) can improve performance across all models and tasks on both test splits. Furthermore, geographic batching leads to a good trade-off between regularization and specialization during TTT, which can be beneficial for long-tail distributions. Our dataset, code, and visualization tool are linked on the project page.

Sources

Related papers