EarthScape: A Multimodal Benchmark for Geologic Mapping and Surface-Aware Learning under Geographic Domain Shift

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "EarthScape: A Multimodal Benchmark for Geologic Mapping and Surface-Aware Learning under Geographic Domain Shift".

Jane: Here's the gist of the paper. EarthScape introduces an AI-ready multimodal dataset for surficial geologic mapping that integrates digital elevation models, aerial imagery, multi-scale terrain features, and hydrologic and infrastructure vector data within a unified pipeline The experiments show that terrain features provide the most reliable predictive signal while raw spectral and elevation inputs degrade substantially under cross-region evaluation Dataset Composition and Structure EarthScape is designed as a multimodal dataset for SG mapping, integrating publicly available RGB and near-infrared (NIR) imagery, digital elevation models (DEM), DEM-derived terrain features at multiple scales, and transportation and hydrological vector data into a unified framework The dataset includes eight high-resolution (1:24,000-scale) SG maps covering two areas in the central United States Each patch is 256×256 pixels with 38 co-registered channels, including the mask, RGB+NIR imagery, DEM, multi-scale terrain derivatives, and binary hydrography and infrastructure layers Feature Engineering for Geomorphic Context The dataset includes five DEM-derived terrain features widely used in geomorphometry, such as Slope (S), Profile curvature (PrC) and Planform curvature (PlC), Elevation percentile (EP), and Standard deviation of slope (SDS) These features are computed at multiple spatial scales to capture hierarchical surface structure, with the native DEM downsampled to five additional resolutions following a roughly logarithmic progression The terrain features provide the most discriminative and robust representation across all experiments, as seen in the strongest configuration of multi-scale, multimodal input Performance Benchmarks Across Modalities and Fusion Strategies Experiments establish reproducible baselines across unimodal, multi-scale, and multimodal configurations to evaluate fusion strategies Single-modality experiments show that terrain features provide the strongest overall performance, with EP achieving the highest in-domain F1 score (0.651) Multi-scale and multimodal configurations improve classwise performance for both encoders, with early channel stacking consistently yielding the strongest results Domain Shift and Generalization Analysis The dataset spans two disjoint regions in Kentucky, providing a natural geographic partition for analyzing cross-region variation Maximum Mean Discrepancy (MMD) analysis quantifies distributional differences between patch-level feature summaries, revealing measurable domain shift for modalities like RGB (0.365) and DEM (0.832) Modalities with smaller input distribution differences tend to transfer more reliably across regions, indicating that shape-based features provide more region-invariant cues than raw appearance or elevation Class-Level Behavior and Model Comparison Class-level AUC varies substantially across units and cannot be explained by frequency alone, with rare units surprisingly being separable in benchmarks ResNeXt-50 performs best on af1, Qal, Qaf, and Qat, whereas ViT-B/16 achieves higher scores on Qc, Qca, and Qr Overall performance is shaped by the interaction of frequency, patch-level mixing, spatial footprint, and scale-dependent expression of each class EarthScape is designed as a living dataset that is extensible in both geographic and modality space, supporting continuous expansion through versioned releases While geographically compact, it is unusually deep with 38 co-registered channels per patch, presenting a distinctive multimodal learning challenge Ongoing work includes expanding coverage and incorporating globally available features to strengthen EarthScape as a benchmark for multimodal fusion and domain adaptation in geospatial AI The code and dataset are publicly available, fostering reproducible research across disciplines Acknowledgements This work is partially supported by the National Science Foundation (NSF) under Grant number 2344533, and the Kentucky Geological Survey The code used for data preprocessing, patch extraction, model training, and evaluation is publicly available at https://github.com/masseygeo/earthscape The full EarthScape dataset is publicly available at https://uknowledge.uky.edu/kgs data/16 References

1: Irasema Alcantara-Ayala, Geomorphology, natural hazards, vulnerability and prevention of natural disasters in developing countries, Geomorphology, 47(2-4):107–124, 2002

3: Guillaume Astruc et al., Omnisat: Self-supervised modality fusion for earth observation, In European Conference on Computer Vision, pages 409–427.

Tom: Springer, 2024

78: Zhitong Xiong et al., Neural plasticity-inspired multimodal foundation model for earth observation, arXiv preprint arXiv:2403.15356, 2024

79: Yiming Zhou et al., A hyper-pixel-wise contrastive learning augmented segmentation network for old landslide detection using high-resolution remote sensing images and digital elevation model data, arXiv preprint arXiv:2308.01251, 2023

81: Boran Han et al., Bridging remote sensors with multisensor geospatial foundation models, In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 27852–27862, 2024

93: Ashish Vaswani et al., Attention is all you need, Advances in neural information processing systems, 30, 2017

94: Tsung-Yi Lin et al.

Title and authors: Tom: That title says it all—it’s about using this EarthScape dataset as a benchmark for surface-aware learning when the geographic area changes from one place to another. It tackles the problem of domain shift, which is huge for real-world applications.

Jane: Exactly, and before we get into the details, let's look at who wrote this paper. The authors are Matthew A. Massey from the Kentucky Geological Survey and Nusrat Munia and Abdullah-Al-Zubaer Imran from the Department of Computer Science at University of Kentucky.

Lu: It’s good to see researchers from different fields collaborating on something like this, trying to bridge the gap between traditional geomorphology and modern machine learning techniques.

Meng: I just hope their system actually runs efficiently when you try to deploy it for, say, real-time infrastructure monitoring, because I've seen some datasets that look great on paper but choke in production.

Lalam: The paper is designed as a multimodal dataset for SG mapping that integrates digital elevation models, aerial imagery, multi-scale terrain features, and hydrologic and infrastructure vector data within a unified pipeline.

The paper's summary: Tom: The main point here is that they tested different ways of combining the data—single modality versus multi-scale versus multimodal—and they found that those detailed terrain features are the most reliable signal for making predictions.

Jane: So, what does that mean in practice? It means if you want to predict something about the land, like identifying a specific type of slope or feature, using just the raw satellite picture or just the elevation map isn't as good as looking at all those derived terrain features together.

Lu: The paper shows that when they use these multi-scale terrain derivatives—things like slope and curvature at different spatial scales—the performance is the strongest across all their experiments.

Meng: That makes sense, because raw spectral data or just elevation can be misleading if you're looking at a region different from where the model was trained.

Lalam: The experiments show that terrain features provide the most reliable predictive signal while raw spectral and elevation inputs degrade substantially under cross-region evaluation.

The paper's improvements: Tom: The authors suggest a few ways to make this system even better, like using early channel stacking when combining the data, which they found consistently yielded the strongest results across their tests.

Jane: That's a practical tip for anyone building a multimodal model—start by putting all your inputs together at the beginning of the network before it starts processing everything. It seems to work well for this type of geospatial data.

Lu: They also highlight that they use five different resolutions for their native digital elevation model, following a roughly logarithmic progression, to capture that hierarchical surface structure across six spatial scales in total.

Meng: Capturing hierarchy is important because it means the AI isn't just seeing the big picture; it's also paying attention to smaller details, which is something we always struggle with when training models.

Lalam: The paper shows they compute five DEM-derived terrain features widely used in geomorphometry, such as slope (S), profile curvature (PrC) and planform curvature (PlC), elevation percentile (EP), and standard deviation of slope (SDS).

Conclusion: Tom: So, wrapping this up, EarthScape is a deep dataset with thirty-eight co-registered channels per patch, which presents a significant multimodal learning challenge. The authors emphasize that it’s an extensible dataset you can keep adding to geographically and modality space.

Jane: It’s a powerful resource because it forces researchers to move away from just looking at one type of data and toward understanding how different data types interact spatially and temporally across different regions.

Lu: The main implication is that shape-based features, like the terrain derivatives, are more region-invariant cues than raw appearance or elevation when dealing with domain shift challenges.

Meng: From an engineering standpoint, they are providing a standardized way to compare fusion strategies that have been hard to reproduce before. It gives us something concrete to test against instead of just guessing what works.

Lalam: EarthScape is designed as a living dataset that is extensible in both geographic and modality space, supporting continuous expansion through versioned releases, and the code and dataset are publicly available for reproducible research across disciplines.