deSEO: Physics-Aware Dataset Creation for High-Resolution Satellite Image Shadow Removal

arXiv:2605.03610 · cs.CV, eess.IV · Submitted 2026-05-05 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom, and with me are Jane, Lu, senior AI researcher at Tsinghua, Meng, lead engineer at a mysterious AI startup and Lalam, the in-house Large Language Model.

Jane: Today's paper: "deSEO: Physics-Aware Dataset Creation for High-Resolution Satellite Image Shadow Removal".

Tom: The gist: deSEO introduces a geometry-aware and physics-informed methodology that derives paired supervision for satellite shadow removal from the S-EO shadow detection dataset through a fully replicable pipeline.

Jane: First, who's behind it and why it matters.

Paper summary: Tom: So, to wrap up, this paper introduces deSEO as a physics-aware methodology that creates paired supervision for satellite shadow removal directly from the S-EO shadow detection dataset through a fully replicable pipeline.

Jane: It’s important to remember that this wasn't just about applying an existing model; they built a whole process for generating the training data itself, which is what makes it unique.

Lu: The core contribution is twofold, creating a geometry-aware pipeline for pairing samples and then developing a deshadowing model that incorporates residual translation, perceptual objectives, and mask-constrained adversarial learning.

Meng: The results show that the full model achieves the best balance across metrics like SSIM and LPIPS, meaning it handles shadows visually well while maintaining good structural fidelity on held-out scenes.

Tom: And their ablation study confirmed that the digital surface model input is a dominant factor, showing how much it actually helps when you remove it versus when you don't.

Jane: Ultimately, deSEO provides the first reproducible, geometry-aware paired dataset and baseline for shadow removal in satellite Earth observation.

Lu: It establishes a framework upon which more robust and generalisable deshadowing methods can be developed because of this structured data creation approach.

Meng: For engineers on the ground, it means they have a solid foundation to start improving their models without having to reinvent the wheel on how to handle these acquisition variations.

Tom: This paper is about taking existing detection-focused resources and engineering them into datasets that are explicitly designed for weakly supervised deshadowing in high-resolution satellite imagery.

Conclusion: Tom: So, deSEO is basically about fixing how we train AI models to get rid of shadows in high-resolution satellite pictures by creating a really good training set that respects the physics of light.

Jane: It takes this existing shadow detection dataset and builds a whole new pipeline to create paired samples, which means the AI learns exactly what to remove.

Lu: The authors used something called geometry-aware methods and LoFTR registration to match shadowed images with shadow-free ones using temporal and geometric filters. It’s a very structured way of making training data.

Meng: From an engineering standpoint, the method has to handle all that off-nadir viewing angles, which is tricky because the sensor isn't always looking straight down at the Earth.

Lalam: The underlying vision model combines residual translation with perceptual objectives and mask-constrained adversarial learning, which helps it focus only on those shadow areas it needs to clean up.

Tom: So what does this mean? It means we finally have a reproducible way to create data for shadow removal that accounts for how the sensor is angled, which has been a huge problem before.

Jane: It’s about moving past just using pretty picture matching and making sure the model understands the actual geometry of light hitting the scene.

Lu: The authors say this is the first fully replicable pipeline for this kind of paired supervision from that specific S-EO dataset, which opens up a lot of doors for future research.

Meng: I think what's practical here is that if we can generate training data this reliably, we can build much more robust tools for things like building extraction or terrain modeling.

Tom: Exactly. It sets a new standard for how we approach creating supervision signals in these kinds of complex remote sensing tasks.

Jane: And it shows us that combining physics knowledge, like light transport theory, with deep learning techniques really helps when dealing with real-world imagery challenges.

Lu: This work gives us a solid baseline for future AI research on shadow removal because now we have a better way to test and compare different approaches.

Meng: So the next big question is whether this structured dataset creation approach will translate directly into better performance in real-world applications, or if there are still major hurdles when deploying these models.

Austrian Institute of Technology · University of Cambridge · University of Wurzburg

cs.CV, eess.IV

Submitted: 2026-05-05

Updated: 2026-05-05

Comments: 8 pages, 6 figures, 5 tables. Accepted in the annals track at the ISPRS 2026 Congress. Code and materials: https://github.com/AIT-Assistive-Autonomous-Systems/deSEO

DOI: 10.5194/isprs-annals-XI-3-2026-11-2026

Code: https://github.com/AIT-Assistive-Autonomous-Systems/deSEO

License: http://creativecommons.org/licenses/by-sa/4.0/

Importance score: 89/100

The gist: The gist: deSEO introduces a geometry-aware and physics-informed methodology that derives paired supervision for satellite shadow removal from the S-EO shadow detection dataset through a fully

Key concepts

Paired Supervision: This refers to creating training examples where the input image (shadowed) is explicitly matched with a corresponding output (shadow-free). deSEO generates these pairs from a single detection dataset by intelligently selecting temporally and geometrically related images, allowing deep learning models to learn shadow removal more effectively.
Geometry Extraction: This step calculates crucial viewing parameters like off-nadir angle and sun elevation for every image. This geometric information is essential because it helps the system understand how shadows are cast based on the satellite's exact position relative to the terrain and sun, enabling better alignment.
LoFTR-RANSAC Registration: This is a technique used for precise image matching between different views or dates. LoFTR (local feature transformer) finds similar features across images, and RANSAC (Random Sample Consensus) refines these matches by filtering out outliers. This ensures that when pairing images, the geometric correspondence between them is highly accurate.
Validity Masking: A binary mask is created for each image to mark regions where the alignment between paired samples is reliable. The neural network training only focuses on these reliably aligned areas, which helps the model learn to remove shadows even when there are slight geometric differences, such as off-nadir viewing angles.

Terminology

Adaptation — 1× in this paper · explained in 1 episode
Fidelity — 1× in this paper · explained in 2 episodes

Summary

The gist: deSEO introduces a geometry-aware and physics-informed methodology that derives paired supervision for satellite shadow removal from the S-EO shadow detection dataset through a fully replicable pipeline.

Introduction

Shadows cast by terrain and tall structures remain a major obstacle for high-resolution satellite image analysis, degrading classification, detection, and 3D reconstruction performance The abundance of Earth Observation (EO) data in combination with advanced machine learning techniques is currently revolutionising the field of remote sensing Physics-based methods, grounded in light transport theory, have long provided solutions for shadow detection and removal Existing shadow-aware EO datasets, such as the Aerial Imagery Shadow Detection (AISD) dataset (Luo et al., 2020) and the CUHK-Shadow dataset (Hu et al., 2021), are primarily designed for shadow detection, rather than their removal More broadly, current resources seldom control for seasonal shifts, viewing geometry differences, or illumination changes, all of which introduce confounding factors that hinder the training and fair evaluation of shadow-removal models in high-resolution satellite imagery.

Methodology

The deSEO methodology is a preprocessing and training pipeline that transforms the S-EO dataset into a dataset suitable for weakly supervised deep learning of shadow removal in high-resolution satellite imagery The pipeline leverages S-EO’s geometry-aware design and multi-temporal, multi-angle acquisitions to create paired samples that support data-driven deshadowing under realistic acquisition variability.

The deSEO pipeline consists of several steps:

Data Organisation and Matching, where each scene is associated with RGB images, binary shadow masks, and acquisition metadata.
Geometry Extraction, which derives key viewing and solar geometry (i.e., off-nadir angle, look azimuth, sun elevation, sun azimuth) for further downstream filtering.
Window Sampling, where a fixed number of K random windows of size C × C px is drawn using a reproducible seed.
Per-window Target Selection and Pairing, which involves selecting the target date t⋆ = arg max t ρ(w, t) = arg min t (1 − ρ(w, t)) based on the clear fraction ρ(w, t).
Candidate Pairing among dates i ≠ t⋆ is accepted only if it satisfies temporal, geometric, and data-quality constraints such as Footprint overlap exceeding τIoU=0.7 and View-geometry proximity tolerances.
DSM Association & Pair Record Creation, where a DSM is linked to each accepted pair to enable downstream data processing.

Geometric Harmonisation and Validity Masking

A practical challenge addressed is the prevalence of off-nadir acquisitions, which are handled by using RPC metadata to align and normalise geometry across dates and viewing directions in sensor space through the use of feature-based aligners. Pairwise Registration uses the detector-free local feature transformer (LoFTR) framework, followed by bidirectional affine estimation with RANSAC to estimate transformations. To ensure supervision is applied only where cross-view correspondence is well-defined, a binary validity mask M ∈ 0, 1 H×W is propagated alongside each training pair after warping.

Neural Network Architecture

The deshadowing model consists of a U-Net generator (Ronneberger et al., 2015) that predicts a residual deshadowing correction and a PatchGAN-style discriminator (Isola et al., 2017) enhanced by a soft, mask-driven attention prior. The generator G is realized by a U-Net with skip connections, employing optional spectral normalisation and non-local self-attention layers to improve context modelling.

The generator objective LG combines reconstruction, perceptual, and color-consistency losses applied only on valid correspondences to ensure stability under weak alignment. The discriminator is enhanced with a soft shadow attention mask (SSAM) that focuses the adversarial signal on regions transitioning from shadow to non-shadow.

Results and Ablation Study

The results show that the full model achieves the best balance across metrics, with the highest SSIM and the lowest VGG19 and LPIPS. The ablation study confirms that DSM input is a dominant factor, causing the largest degradation when removed. Furthermore, removing self-attention or gamma features leads to moderate drops in structural fidelity and perceptual quality. The model consistently identifies and reduces shadows under diverse illumination and viewing conditions, providing a basis for further improvement.

Conclusion

deSEO establishes a first reproducible, physics-aware dataset creation methodology for weakly supervised deshadowing in high-resolution satellite imagery. This framework provides a foundation upon which more robust and generalisable deshadowing methods can be developed. deSEO establishes a first reproducible, physics-aware dataset creation methodology for weakly supervised deshadowing in high-resolution satellite imagery.

Acknowledgments

This work was carried out within the SAFIR research project funded by the Austrian Research Promotion Agency (FFG) as part of the Research, Technology & Innovation (RTI) initiative ‘Digitaler Zwilling Osterreich’.

References

Dare, P. M. (2005). ‘Shadow analysis in high-resolution satellite imagery of urban areas’. In: Photogramm. Eng. Remote Sens. 71.2, pp. 169–177

Dong, X., Cao, J. and Zhao, W. (2023). ‘A review of research on remote sensing images shadow detection and application to building extraction’. In: Eur. J. Remote Sens. 57.1, p. 2293163 DOI: 10.1080/22797254

Fischler, M. A. and Bolles, R. C. (1981). ‘Random Sample Consensus: A Paradigm for Model Fitting with Applications to Image Analysis and Automated Cartography’. In: Communications of the ACM 24.6, pp. 381–395 DOI: 10.1145/358669.358692

Giles, P. T. (2001). ‘Remote sensing and cast shadows in mountainous terrain’. In: Photogramm. Eng. Remote Sens. 67.7, pp. 833–840

Hu, X. et al. (2021). ‘Revisiting Shadow Detection: A New Benchmark Dataset for Complex World’. In: IEEE Transactions on Image Processing 30, pp. 1925–1934 DOI: 10.1109/tip.2021.3049331

Isola, P. et al. (2017). ‘Image-to-image translation with conditional adversarial networks’. In: IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), pp. 5967–5976 DOI: 10.1109/cvpr.2017.632

Johnson, J., Alahi, A. and Fei-Fei, L. (2016). ‘Perceptual Losses for RealTime Style Transfer and Super-Resolution’. In: Eur. Conf. Comput. Vis. (ECCV), pp. 694–711 DOI: 10.1007/978-3-319-46475-6 43

Jovhari, N. et al.

Improvements for AI systems

The deSEO pipeline can be used to transform detection-oriented EO datasets into paired resources for weakly supervised deshadowing through principled geometric filtering and rearXiv:2605.03610v1 [cs.CV] 5 May 2026. This enables the creation of a first reproducible, geometry-aware paired dataset and baseline for shadow removal in satellite Earth observation by selecting a minimally shadowed acquisition as a weak reference and pairs it with shadowed counterparts using temporal and geometric filtering, Jacobian-based orientation normalisation, and LoFTR–RANSAC registration.
The proposed deshadowing model can be enhanced by incorporating DSM information to improve reconstruction quality. The paper states that the "DSM input, perceptual supervision, and stabilising regularisation (HV, spectral norm, attention) contribute jointly to the model’s ability to preserve both radiometric and perceptual fidelity while performing meaningful shadow reduction."
The system can be trained using a geometry-aware loss function that restricts supervision to reliable correspondences. The architecture utilizes registration-driven validity masks to restrict supervision to reliable correspondences, meaning the L1 loss of the pix2pix generator and the discriminator inputs are restricted to the valid subset by masking and renormalisation.
The model can be made more robust against viewpoint variability by employing a geometry-aware discriminator. The paper describes a soft shadow attention mask (SSAM) that focuses the adversarial signal on regions transitioning from shadow to non-shadow, which emphasises areas of shadow disappearance and complements reconstruction losses by ensuring the model learns illumination-consistent corrections.
The system can be optimized for high-resolution detail preservation using self-attention mechanisms within the generator. The architecture employs selfattention at intermediate feature resolutions best preserves finegrained structure in shadow removal, which is crucial for balancing local detail with contextual consistency and avoiding over-smoothing at high resolutions.

Abstract

Shadows cast by terrain and tall structures remain a major obstacle for high-resolution satellite image analysis, degrading classification, detection, and 3D reconstruction performance. Public resources offering geometry-consistent paired shadow/shadow-free satellite imagery are essentially missing, and most Earth-observation datasets are designed for shadow detection or 3D modelling rather than removal. Existing deep shadow-removal datasets either target ground-level or aerial scenes or rely on unpaired and weakly supervised formulations rather than explicit satellite pairs. We address this gap with deSEO, a geometry-aware and physics-informed methodology that, to the best of our knowledge, is the first to derive paired supervision for satellite shadow removal from the S-EO shadow detection dataset through a fully replicable pipeline. For each tile, deSEO selects a minimally shadowed acquisition as a weak reference and pairs it with shadowed counterparts using temporal and geometric filtering, Jacobian-based orientation normalisation, and LoFTR-RANSAC registration. A per-pixel validity mask restricts learning to reliably aligned regions, enabling supervision despite residual off-nadir parallax. In addition to this paired dataset, we develop a DSM-aware deshadowing model that combines residual translation, perceptual objectives, and mask-constrained adversarial learning. In contrast, a direct adaptation of a UAV-based SRNet/pix2pix architecture fails to converge under satellite viewpoint variability. Our model consistently reduces the visual impact of cast shadows across diverse illumination and viewing conditions, achieving improved structural and perceptual fidelity on held-out scenes. deSEO therefore provides the first reproducible, geometry-aware paired dataset and baseline for shadow removal in satellite Earth observation.