How Much of a Model Do We Need? Redundancy and Slimmability in Remote Sensing Foundation Models
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: I'm Tom.
Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.
Tom: We're looking at the paper: "How Much of a Model Do We Need? Redundancy and Slimmability in Remote Sensing Foundation Models".
Jane: Here's the gist of the paper. Large-scale foundation models (FMs) in remote sensing (RS) are developed following paradigms established in computer vision (CV), yet the validity of transferring CV scaling laws to RS has not been systematically examined The single most informative sentence about the paper's subject and finding is that RS FMs enter an overparameterized regime at substantially smaller scales than their CV counterparts, with task-relevant information encoded redundantly across model dimensions Hypothesis and Methodology The authors hypothesize that RS FMs enter an overparameterized regime at substantially smaller scales than their CV counterparts, with task-relevant information encoded redundantly across model dimensions To test this hypothesis, they apply post-hoc slimmability, uniform width reduction of pretrained encoder transformer blocks, as a tool to measure representational redundancy across eight state-of-the-art RS FMs on classification, segmentation, and change detection tasks They conduct a comprehensive empirical study across these eight RS FMs ranging from 23 M to 631 M parameters, evaluating post-hoc slimmability on seven downstream datasets spanning scene classification, semantic segmentation, and change detection Performance Comparison RS FMs retain 69 % to 109 % relative accuracy on RS datasets under aggressive width reduction In contrast, masked autoencoder (MAE) and DINOv2 pretrained on natural images (denoted as CV MAE and CV DINOv2) degrade sharply on ImageNet subsets of matched class count over the same range of computational requirements Post-hoc slimming to 1 % of their computational budget retains 69 % to 100 % accuracy across classification datasets, with the majority of models retaining above 80 %, and that this robustness persists for other tasks Mechanistic Insights Mechanistic analyses such as feature correlation, explained variance, and effective dimensionality indicate that task-relevant variance concentrates in few principal components and is redundantly encoded across model dimensions The study shows that learned slimmable training improves over post-hoc slimmability for contrastive objectives, while reconstruction-based objectives do not benefit from current slimmable training protocols Furthermore, simultaneous slimming of both layer types proves superior across all evaluated models, enabling compression to 1 % relative compute while exhibiting the most gradual performance degradation curves Implications for Deployment Post-hoc slimming from 5 % to 20 % of the full compute budget frequently maintains over 80 % relative accuracy without any retraining Computational budget can therefore be treated as a continuous deployment variable, with task complexity determining the effective lower bound Simpler tasks such as single-label scene classification tolerate more aggressive slimming than demanding ones such as multi-label scene classification Model Family Strategies The analysis reveals distinct correlation shapes remain consistent across both model sizes and datasets, indicating that these patterns are fundamental characteristics of training paradigms rather than task-specific adaptations DOFA models exhibit a characteristic correlation shape across all datasets: highly organized feature correlations at small scales transitioning to near-independence at full scale TerraMind-1.0 models demonstrate identical correlation shapes across both base and large sizes, characterized by smooth and mostly monotonic decrease from moderate to low correlations Prithvi-EO-2.0 models exhibit the distinctive non-monotonic U-shaped correlation pattern across both Prithvi-EO-2.0 (300M) and Prithvi-EO-2.0 (600M) and all datasets The findings establish post-hoc slimming as a practical deployment strategy for resource-constrained RS applications and as a diagnostic tool for representational redundancy in RS FMs The study confirms that distributed encoding across moderately-dimensional subspaces enables robust slimmability, while the most aggressive dimensionality concentration at low compute (DOFA) re19 References How Much of a Model Do We Need?
Tom: Redundancy and Slimmability in Remote Sensing Foundation Models How Much of a Model Do We Need?
Jane: Redundancy and Slimmability in Remote Sensing Foundation Models How Much of a Model Do We Need?
Tom: Redundancy and Slimmability in Remote Sensing Foundation Models How Much of a Model Do We Need?
Paper summary: Jane: Okay, so they test this idea by applying post-hoc slimmability and uniform width reduction to eight state-of-the-art remote sensing foundation models across classification, segmentation, and change detection tasks.
Tom: They look at models ranging from twenty-three million to six hundred thirty-one million parameters. What they found is that these remote sensing models keep sixty nine percent up to one hundred nine percent relative accuracy on remote sensing datasets even when you aggressively reduce their width.
Lu: That retention rate, especially when compared against computer vision baselines like the CV MAE and DINOv2—which only retain thirty to thirty five percent on ImageNet-ten and one hundred—really highlights the difference in how these models are structured for remote sensing.
Meng: So if we can get this level of retention with aggressive width reduction, that’s a massive win for deploying these models in real-world scenarios where computational budget is tight.
Lalam: The paper also looked at the performance trade-offs between post-hoc slimmability and learned slimmable training methods, finding that learned training helps some models but not others depending on what objective they are using.
Jane: It seems like the study confirms that for remote sensing, there’s a lot of robustness to reducing compute, which is exactly what practitioners need to know when thinking about deployment.
Tom: It shows that post-hoc slimming down to one percent of the full compute budget still keeps sixty nine percent up to one hundred percent accuracy across classification datasets, and most models stay above eighty percent.
Lu: And they provide some mechanistic evidence too, using feature correlation and explained variance analysis to show that task-relevant variance concentrates in a few principal components but is distributed redundantly across different dimensions.
Meng: That redundancy is what makes it work; the important parts aren't just in one layer or one dimension, they are spread out enough that you can cut some away.
Lalam: They also found that simultaneous slimming of both the feature extraction layers and the feed forward layers proves superior across all eight models, allowing compression to one percent relative compute while showing the most gradual performance degradation curves.
Conclusion: Jane: So, looking at the whole paper "How Much of a Model Do We Need? Redundancy and Slimmability in Remote Sensing Foundation Models," what does this all mean for us outside of just shrinking models?
Tom: The main conclusion is that post-hoc slimming is a practical deployment strategy for remote sensing applications and it serves as a diagnostic tool to check for representational redundancy within these foundation models.
Lu: It confirms that distributing the encoding across moderately dimensional subspaces allows for this robust slimmability, and they also found that the most aggressive dimensionality concentration at low compute, which they call DOFA models, shows this behavior.
Meng: For me, it means we can treat the computational budget as a continuous deployment variable instead of a fixed number; the complexity of the task determines where your effective lower bound is.
Lalam: It also gives us some guidance on when to apply aggressive slimming; simpler tasks like single-label scene classification can handle more aggressive slimming than more demanding tasks like multi-label scene classification.
Jane: So, in simple terms, this paper suggests that remote sensing models are surprisingly robust to compression because the important information isn't stuck in one spot but is spread out everywhere.
Tom: It’s about understanding that distributed encoding across different dimensions enables robust slimmability in remote sensing foundation models.
BIFOLD / TU Berlin
cs.CV
Submitted: 2026-01-30
Updated: 2026-11-11
Comments: Accepted at Advances in Neural Information Processing Systems (NeurIPS) 2026. Code is available at https://github.com/hackelle/slimmable-remote-sensing-foundation-models
Code: https://github.com/ServiceNow/geo-bench
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 90/100
The gist: Large-scale foundation models (FMs) in remote sensing (RS) are developed following paradigms established in computer vision (CV)…
Key concepts
- Overparameterized Regime
- This refers to a state where a model has far more parameters than necessary to solve the given task. The paper suggests that RS FMs enter this regime at smaller scales than CV models, meaning they contain significant redundancy, allowing them to perform well even when significantly reduced in size.
- Post-hoc Slimmability
- This is a technique where researchers reduce the computational budget of a pre-trained model after it has been trained. The study used this to measure how much information can be removed while keeping performance high, revealing that RS models are robust to aggressive slimming.
- Representational Redundancy
- This concept describes how task-relevant information is stored in multiple, overlapping ways across the model's dimensions. The analysis showed that crucial information is concentrated in a few principal components but is spread redundantly, which enables the model to be effectively compressed without losing performance.
- Continuous Deployment Variable
- This frames computational budget not as a fixed constraint but as something that can be adjusted during deployment. The findings imply that task complexity determines the minimum required compute, allowing researchers to treat compute reduction as a flexible parameter for different application needs.
Terminology
Summary
Large-scale foundation models (FMs) in remote sensing (RS) are developed following paradigms established in computer vision (CV), yet the validity of transferring CV scaling laws to RS has not been systematically examined The single most informative sentence about the paper's subject and finding is that RS FMs enter an overparameterized regime at substantially smaller scales than their CV counterparts, with task-relevant information encoded redundantly across model dimensions
Hypothesis and Methodology
The authors hypothesize that RS FMs enter an overparameterized regime at substantially smaller scales than their CV counterparts, with task-relevant information encoded redundantly across model dimensions To test this hypothesis, they apply post-hoc slimmability, uniform width reduction of pretrained encoder transformer blocks, as a tool to measure representational redundancy across eight state-of-the-art RS FMs on classification, segmentation, and change detection tasks They conduct a comprehensive empirical study across these eight RS FMs ranging from 23 M to 631 M parameters, evaluating post-hoc slimmability on seven downstream datasets spanning scene classification, semantic segmentation, and change detection
Performance Comparison
RS FMs retain 69 % to 109 % relative accuracy on RS datasets under aggressive width reduction In contrast, masked autoencoder (MAE) and DINOv2 pretrained on natural images (denoted as CV MAE and CV DINOv2) degrade sharply on ImageNet subsets of matched class count over the same range of computational requirements Post-hoc slimming to 1 % of their computational budget retains 69 % to 100 % accuracy across classification datasets, with the majority of models retaining above 80 %, and that this robustness persists for other tasks
Mechanistic Insights
Mechanistic analyses such as feature correlation, explained variance, and effective dimensionality indicate that task-relevant variance concentrates in few principal components and is redundantly encoded across model dimensions The study shows that learned slimmable training improves over post-hoc slimmability for contrastive objectives, while reconstruction-based objectives do not benefit from current slimmable training protocols Furthermore, simultaneous slimming of both layer types proves superior across all evaluated models, enabling compression to 1 % relative compute while exhibiting the most gradual performance degradation curves
Implications for Deployment
Post-hoc slimming from 5 % to 20 % of the full compute budget frequently maintains over 80 % relative accuracy without any retraining Computational budget can therefore be treated as a continuous deployment variable, with task complexity determining the effective lower bound Simpler tasks such as single-label scene classification tolerate more aggressive slimming than demanding ones such as multi-label scene classification
Model Family Strategies
The analysis reveals distinct correlation shapes remain consistent across both model sizes and datasets, indicating that these patterns are fundamental characteristics of training paradigms rather than task-specific adaptations DOFA models exhibit a characteristic correlation shape across all datasets: highly organized feature correlations at small scales transitioning to near-independence at full scale TerraMind-1.0 models demonstrate identical correlation shapes across both base and large sizes, characterized by smooth and mostly monotonic decrease from moderate to low correlations Prithvi-EO-2. p. 8 0 (large) shows a saturating rather than monotonic structure. Feature correlation declines from moderate values (∼0. 0 (600M) exhibits a qualitatively distinct structure across
all three analyses. Feature correlation follows a non-monotonic trajectory across the compute range
(Figure 4c), and the EVR slope shows a corresponding non-monotonic pattern (Figure 6). As shown in Table 1, widely used RS FMs span
multiple orders of magnitude in parameters and computational budgets. For instance, Prithvi [19, 34]
leverages Sentinel-2 and Landsat time-series data, while Satlas [1] introduces 300M+ annotations
for supervision, and SSL4EO-S12 [35] provides vision transformer (ViT) encoders pretrained via
contrastive objectives on a large-scale multi-seasonal Sentinel-2 dataset.Where the paper says it
Where the paper says it
Where the paper says it
Conclusion
The findings establish post-hoc slimming as a practical deployment strategy for resource-constrained RS applications and as a diagnostic tool for representational redundancy in RS FMs The study confirms that distributed encoding across moderately-dimensional subspaces enables robust slimmability, while the most aggressive dimensionality concentration at low compute (DOFA) re19
References
How Much of a Model Do We Need? Redundancy and Slimmability in Remote Sensing Foundation Models
How Much of a Model Do We Need? Redundancy and Slimmability in Remote Sensing Foundation Models
How Much of a Model Do We Need? Redundancy and Slimmability in Remote Sensing Foundation Models
How Much of a Model Do We Need? Redundancy and Slimmability in Remote Sensing Foundation Models
Technical appendices and supplementary material
Feature Correlation at different Model Scales
TerraMind-1.0 (base) and large (Figure 15c and 15d)
Geobench: Toward foundation models for earth monitoring
How Much of a Model Do We Need? Redundancy and Slimmability in Remote Sensing Foundation Models
<ref:2601.22841#pg4>RS FMs @ 4 RS datasets (avg. p. 1 RS FMs @ 4 RS datasets (avg. )
CV DINOv2 @ ImageNet-10
CV MAE @ ImageNet-10
CV MAE @ ImageNet-100
RS FMs @ m-eurosat
CV MAE @ m-eurosat 0. Legends formatted
as: “model(s) @ evaluation data”. a) Relative retention rate: RS FMs averaged across four RS
classification datasets maintain over 85 % relative accuracy at 1 % compute budget, whereas CV
DINOv2 and CV MAE retain on average only 30 % to 35 % on ImageNet-10 and a CV MAE only
12 % on ImageNet-100 at the same compute budget. 575
mAPµ
Rel. Compute Requirement
(b)
Figure 3: k-NN classification performance with respect to their relative compute requirements of
a MAE-based and a MoCo-based trained model from scratch, each once with and once without
slimmable training on: a) m-eurosat; and b) m-bigearthnet. Compute Requirement
(b)
0. 40
(c)
Figure 4: Mean Feature Correlation of the test split of m-eurosat and m-bigearthnet from the geobench
benchmark collection with respect to their relative compute requirements on pretrained RS FMs:
a) DOFA (large); b) TerraMind-1. As shown in Table 1, widely used RS FMs span
multiple orders of magnitude in parameters and computational budgets. For instance, Prithvi [19, 34]
leverages Sentinel-2 and Landsat time-series data, while Satlas [1] introduces 300M+ annotations
for supervision, and SSL4EO-S12 [35] provides vision transformer (ViT) encoders pretrained via
contrastive objectives on a large-scale multi-seasonal Sentinel-2 dataset. Larger models within each architecture (DOFA (large),
Prithvi-EO-2. 0 (600M), TerraMind-1. As shown in Table 1, widely used RS FMs span
multiple orders of magnitude in parameters and computational budgets. For instance, Prithvi [19, 34]
leverages Sentinel-2 and Landsat time-series data, while Satlas [1] introduces 300M+ annotations
for supervision, and SSL4EO-S12 [35] provides vision transformer (ViT) encoders pretrained via
contrastive objectives on a large-scale multi-seasonal Sentinel-2 dataset. 0 (300M)
Prithvi-EO-2. 0 (600M)
SSL4EO ViT-S (Dino)
SSL4EO ViT-S (MoCo)
TerraMind-1. 575
mAPµ
Rel. Compute Requirement
(b)
Figure 3: k-NN classification performance with respect to their relative compute requirements of
a MAE-based and a MoCo-based trained model from scratch, each once with and once without
slimmable training on: a) m-eurosat; and b) m-bigearthnet. 0 (300M)
Prithvi-EO-2. 0 (600M)
SSL4EO ViT-S (Dino)
SSL4EO ViT-S (MoCo)
TerraMind-1. 0 (600M)
SSL4EO ViT-S (Dino)
SSL4EO ViT-S (MoCo)
TerraMind-1. 0 (base)
TerraMind-1. 0 (600M)
SSL4EO ViT-S (Dino)
SSL4EO ViT-S (MoCo)
TerraMind-1. 0 (base)
TerraMind-1. DOFA (base)
DOFA (large)
Prithvi-EO-2. 0 (300M)
Prithvi-EO-2. As shown in Table 1, widely used RS FMs span
multiple orders of magnitude in parameters and computational budgets. For instance, Prithvi [19, 34]
leverages Sentinel-2 and Landsat time-series data, while Satlas [1] introduces 300M+ annotations
for supervision, and SSL4EO-S12 [35] provides vision transformer (ViT) encoders pretrained via
contrastive objectives on a large-scale multi-seasonal Sentinel-2 dataset. 0 (300M)
Prithvi-EO-2. 0 (600M)
SSL4EO ViT-S (Dino)
SSL4EO ViT-S (MoCo)
TerraMind-1. Compute Requirement
(d)
Figure 2: Downstream performance with respect to relative compute requirements of eight pretrained
RS FMs on: a) m-eurosat (single-labels scene classification); b) m-bigearthnet (multi-label scene
classification); c) m-cashew-plant (semantic segmentation); d) OSCD (change detection). Results for
m-brick-kiln, m-so2sat, and m-SA-crop-type are provided in Appendix G. 575
mAPµ
Rel. Compute Requirement
(b)
Figure 3: k-NN classification performance with respect to their relative compute requirements of
a MAE-based and a MoCo-based trained model from scratch, each once with and once without
slimmable training on: a) m-eurosat; and b) m-bigearthnet. Compute Requirement
(b)
0. 40
(c)
Figure 4: Mean Feature Correlation of the test split of m-eurosat and m-bigearthnet from the geobench
benchmark collection with respect to their relative compute requirements on pretrained RS FMs:
a) DOFA (large); b) TerraMind-1. 00
Effective Dimension
(b)
10
0 10
1
0. 00
(c)
Figure 5: Effective Dimension of the test split of four classification datasets from the geobench
benchmark collection with respect to their relative retention rate on pretrained RS FMs: a) DOFA
(large); b) TerraMind-1.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
<ref:2601.
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Post-hoc slimming of remote sensing foundation models can be used as a practical deployment strategy for resource-constrained applications because
RS FMs retain 69 % to 109 % relative accuracy on RS datasets under aggressive width reduction.
-
The effective dimensionality analysis reveals that
task-relevant variance concentrates in few principal components and is redundantly encoded across model dimensions,
suggesting that the most aggressive slimming preserves performance by removing this redundant information. -
Learned slimmable training improves MoCo-based models because it
consistently outperforms regular training at low compute budgets while maintaining equivalent or superior performance at higher scales
on m-eurosat. -
The analysis of feature correlation shows that
distributed encoding across moderately-dimensional subspaces enables robust slimmability,
indicating that the organization of information across scales is determined by the training paradigm rather than parameter count. -
Simultaneous slimming of both FFN and MHSA layers is recommended because it
proves superior across all evaluated models, enabling compression to 1 % relative compute while exhibiting the most gradual performance degradation curves.
Abstract
Large-scale foundation models (FMs) in remote sensing (RS) (denoted as RS FMs) are developed following paradigms established in computer vision (CV), yet the validity of transferring CV scaling laws to RS has not been systematically examined. We hypothesize that RS FMs enter an overparameterized regime at substantially smaller scales than their CV counterparts, with task-relevant information encoded redundantly across model dimensions. To test this hypothesis, we apply post-hoc slimmability, uniform width reduction of pretrained encoder transformer blocks, to measure representational redundancy across ten state-of-the-art RS FMs on classification, segmentation, and change detection. RS FMs retain on average over 85% relative accuracy on RS classification datasets under aggressive width reduction, while masked autoencoder (MAE) and DINOv2 pretrained on natural images (denoted as CV MAE and CV DINOv2) degrade sharply on ImageNet subsets of matched class count over the same range of computational requirements. A CV MAE evaluated directly on the same RS dataset narrows but does not close the gap, indicating that both dataset characteristics and domain-specific pretraining contribute to the differences between the models. Analyses of the representation space of the models indicate that task-relevant variance concentrates in few principal components and is redundantly encoded across model dimensions. We further show that learned slimmable training improves over post-hoc slimmability for contrastive objectives, but reconstruction-based objectives do not benefit consistently from current slimmable training protocols. Our findings establish post-hoc slimming as a practical deployment strategy for resource-constrained RS applications and as a diagnostic tool for representational redundancy in RS FMs. Code is available at https://github.com/hackelle/slimmable-remote-sensing-foundation-models.
Sources
- RingMoE: Mixture-of-Modality-Experts Multi-Modal Foundation Models for Universal Remote Sensing Image Interpretation
- CSMoE: An Efficient Remote Sensing Foundation Model with Soft Mixture-of-Experts
- Understanding and Harnessing Sparsity in Unified Multimodal Models
- Distilling the Knowledge in a Neural Network
- Foundation Models for Generalist Geospatial Artificial Intelligence
- Prithvi-EO-2.0: A Versatile Multi-Temporal Foundation Model for Earth Observation Applications
- Neural Plasticity-Inspired Multimodal Foundation Model for Earth Observation
Related papers
- TEA: Text Encoder Alignment for Robust Concept Erasure in Text-to-Image Models
- Do Egocentric Video-Language Models Capture Both Hand- and Object-Centric Cues?
- Aura: Consistent Multi-Subject Video Generation via VLM-Grounded Semantic Alignment
- PVRF: All-in-one Adverse Weather Removal via Prior-modulated and Velocity-constrained Rectified Flow
- DLWM: Diverse Latent World Models for Efficient Multimodal Reasoning
- Out of Sight, Still in Mind: Token Compression for Omni-LLMs