How Much of a Model Do We Need? Redundancy and Slimmability in Remote Sensing Foundation Models

arXiv:2601.22841 · cs.CV · Submitted 2026-01-30 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "How Much of a Model Do We Need? Redundancy and Slimmability in Remote Sensing Foundation Models".

Jane: Here's the gist of the paper. Large-scale foundation models (FMs) in remote sensing (RS) are developed following paradigms established in computer vision (CV), yet the validity of transferring CV scaling laws to RS has not been systematically examined The single most informative sentence about the paper's subject and finding is that RS FMs enter an overparameterized regime at substantially smaller scales than their CV counterparts, with task-relevant information encoded redundantly across model dimensions Hypothesis and Methodology The authors hypothesize that RS FMs enter an overparameterized regime at substantially smaller scales than their CV counterparts, with task-relevant information encoded redundantly across model dimensions To test this hypothesis, they apply post-hoc slimmability, uniform width reduction of pretrained encoder transformer blocks, as a tool to measure representational redundancy across eight state-of-the-art RS FMs on classification, segmentation, and change detection tasks They conduct a comprehensive empirical study across these eight RS FMs ranging from 23 M to 631 M parameters, evaluating post-hoc slimmability on seven downstream datasets spanning scene classification, semantic segmentation, and change detection Performance Comparison RS FMs retain 69 % to 109 % relative accuracy on RS datasets under aggressive width reduction In contrast, masked autoencoder (MAE) and DINOv2 pretrained on natural images (denoted as CV MAE and CV DINOv2) degrade sharply on ImageNet subsets of matched class count over the same range of computational requirements Post-hoc slimming to 1 % of their computational budget retains 69 % to 100 % accuracy across classification datasets, with the majority of models retaining above 80 %, and that this robustness persists for other tasks Mechanistic Insights Mechanistic analyses such as feature correlation, explained variance, and effective dimensionality indicate that task-relevant variance concentrates in few principal components and is redundantly encoded across model dimensions The study shows that learned slimmable training improves over post-hoc slimmability for contrastive objectives, while reconstruction-based objectives do not benefit from current slimmable training protocols Furthermore, simultaneous slimming of both layer types proves superior across all evaluated models, enabling compression to 1 % relative compute while exhibiting the most gradual performance degradation curves Implications for Deployment Post-hoc slimming from 5 % to 20 % of the full compute budget frequently maintains over 80 % relative accuracy without any retraining Computational budget can therefore be treated as a continuous deployment variable, with task complexity determining the effective lower bound Simpler tasks such as single-label scene classification tolerate more aggressive slimming than demanding ones such as multi-label scene classification Model Family Strategies The analysis reveals distinct correlation shapes remain consistent across both model sizes and datasets, indicating that these patterns are fundamental characteristics of training paradigms rather than task-specific adaptations DOFA models exhibit a characteristic correlation shape across all datasets: highly organized feature correlations at small scales transitioning to near-independence at full scale TerraMind-1.0 models demonstrate identical correlation shapes across both base and large sizes, characterized by smooth and mostly monotonic decrease from moderate to low correlations Prithvi-EO-2.0 models exhibit the distinctive non-monotonic U-shaped correlation pattern across both Prithvi-EO-2.0 (300M) and Prithvi-EO-2.0 (600M) and all datasets The findings establish post-hoc slimming as a practical deployment strategy for resource-constrained RS applications and as a diagnostic tool for representational redundancy in RS FMs The study confirms that distributed encoding across moderately-dimensional subspaces enables robust slimmability, while the most aggressive dimensionality concentration at low compute (DOFA) re19 References How Much of a Model Do We Need?

Tom: Redundancy and Slimmability in Remote Sensing Foundation Models How Much of a Model Do We Need?

Jane: Redundancy and Slimmability in Remote Sensing Foundation Models How Much of a Model Do We Need?

Tom: Redundancy and Slimmability in Remote Sensing Foundation Models How Much of a Model Do We Need?

Paper summary: Jane: Okay, so they test this idea by applying post-hoc slimmability and uniform width reduction to eight state-of-the-art remote sensing foundation models across classification, segmentation, and change detection tasks.

Tom: They look at models ranging from twenty-three million to six hundred thirty-one million parameters. What they found is that these remote sensing models keep sixty nine percent up to one hundred nine percent relative accuracy on remote sensing datasets even when you aggressively reduce their width.

Lu: That retention rate, especially when compared against computer vision baselines like the CV MAE and DINOv2—which only retain thirty to thirty five percent on ImageNet-ten and one hundred—really highlights the difference in how these models are structured for remote sensing.

Meng: So if we can get this level of retention with aggressive width reduction, that’s a massive win for deploying these models in real-world scenarios where computational budget is tight.

Lalam: The paper also looked at the performance trade-offs between post-hoc slimmability and learned slimmable training methods, finding that learned training helps some models but not others depending on what objective they are using.

Jane: It seems like the study confirms that for remote sensing, there’s a lot of robustness to reducing compute, which is exactly what practitioners need to know when thinking about deployment.

Tom: It shows that post-hoc slimming down to one percent of the full compute budget still keeps sixty nine percent up to one hundred percent accuracy across classification datasets, and most models stay above eighty percent.

Lu: And they provide some mechanistic evidence too, using feature correlation and explained variance analysis to show that task-relevant variance concentrates in a few principal components but is distributed redundantly across different dimensions.

Meng: That redundancy is what makes it work; the important parts aren't just in one layer or one dimension, they are spread out enough that you can cut some away.

Lalam: They also found that simultaneous slimming of both the feature extraction layers and the feed forward layers proves superior across all eight models, allowing compression to one percent relative compute while showing the most gradual performance degradation curves.

Conclusion: Jane: So, looking at the whole paper "How Much of a Model Do We Need? Redundancy and Slimmability in Remote Sensing Foundation Models," what does this all mean for us outside of just shrinking models?

Tom: The main conclusion is that post-hoc slimming is a practical deployment strategy for remote sensing applications and it serves as a diagnostic tool to check for representational redundancy within these foundation models.

Lu: It confirms that distributing the encoding across moderately dimensional subspaces allows for this robust slimmability, and they also found that the most aggressive dimensionality concentration at low compute, which they call DOFA models, shows this behavior.

Meng: For me, it means we can treat the computational budget as a continuous deployment variable instead of a fixed number; the complexity of the task determines where your effective lower bound is.

Lalam: It also gives us some guidance on when to apply aggressive slimming; simpler tasks like single-label scene classification can handle more aggressive slimming than more demanding tasks like multi-label scene classification.

Jane: So, in simple terms, this paper suggests that remote sensing models are surprisingly robust to compression because the important information isn't stuck in one spot but is spread out everywhere.

Tom: It’s about understanding that distributed encoding across different dimensions enables robust slimmability in remote sensing foundation models.

BIFOLD / TU Berlin

cs.CV

Submitted: 2026-01-30

Updated: 2026-11-11

Comments: Accepted at Advances in Neural Information Processing Systems (NeurIPS) 2026. Code is available at https://github.com/hackelle/slimmable-remote-sensing-foundation-models

Code: https://github.com/ServiceNow/geo-bench

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 90/100

The gist: Large-scale foundation models (FMs) in remote sensing (RS) are developed following paradigms established in computer vision (CV)…

Key concepts

Overparameterized Regime
This refers to a state where a model has far more parameters than necessary to solve the given task. The paper suggests that RS FMs enter this regime at smaller scales than CV models, meaning they contain significant redundancy, allowing them to perform well even when significantly reduced in size.
Post-hoc Slimmability
This is a technique where researchers reduce the computational budget of a pre-trained model after it has been trained. The study used this to measure how much information can be removed while keeping performance high, revealing that RS models are robust to aggressive slimming.
Representational Redundancy
This concept describes how task-relevant information is stored in multiple, overlapping ways across the model's dimensions. The analysis showed that crucial information is concentrated in a few principal components but is spread redundantly, which enables the model to be effectively compressed without losing performance.
Continuous Deployment Variable
This frames computational budget not as a fixed constraint but as something that can be adjusted during deployment. The findings imply that task complexity determines the minimum required compute, allowing researchers to treat compute reduction as a flexible parameter for different application needs.

Terminology

Summary

Large-scale foundation models (FMs) in remote sensing (RS) are developed following paradigms established in computer vision (CV), yet the validity of transferring CV scaling laws to RS has not been systematically examined The single most informative sentence about the paper's subject and finding is that RS FMs enter an overparameterized regime at substantially smaller scales than their CV counterparts, with task-relevant information encoded redundantly across model dimensions

Hypothesis and Methodology

The authors hypothesize that RS FMs enter an overparameterized regime at substantially smaller scales than their CV counterparts, with task-relevant information encoded redundantly across model dimensions To test this hypothesis, they apply post-hoc slimmability, uniform width reduction of pretrained encoder transformer blocks, as a tool to measure representational redundancy across eight state-of-the-art RS FMs on classification, segmentation, and change detection tasks They conduct a comprehensive empirical study across these eight RS FMs ranging from 23 M to 631 M parameters, evaluating post-hoc slimmability on seven downstream datasets spanning scene classification, semantic segmentation, and change detection

Performance Comparison

RS FMs retain 69 % to 109 % relative accuracy on RS datasets under aggressive width reduction In contrast, masked autoencoder (MAE) and DINOv2 pretrained on natural images (denoted as CV MAE and CV DINOv2) degrade sharply on ImageNet subsets of matched class count over the same range of computational requirements Post-hoc slimming to 1 % of their computational budget retains 69 % to 100 % accuracy across classification datasets, with the majority of models retaining above 80 %, and that this robustness persists for other tasks

Mechanistic Insights

Mechanistic analyses such as feature correlation, explained variance, and effective dimensionality indicate that task-relevant variance concentrates in few principal components and is redundantly encoded across model dimensions The study shows that learned slimmable training improves over post-hoc slimmability for contrastive objectives, while reconstruction-based objectives do not benefit from current slimmable training protocols Furthermore, simultaneous slimming of both layer types proves superior across all evaluated models, enabling compression to 1 % relative compute while exhibiting the most gradual performance degradation curves

Implications for Deployment

Post-hoc slimming from 5 % to 20 % of the full compute budget frequently maintains over 80 % relative accuracy without any retraining Computational budget can therefore be treated as a continuous deployment variable, with task complexity determining the effective lower bound Simpler tasks such as single-label scene classification tolerate more aggressive slimming than demanding ones such as multi-label scene classification

Model Family Strategies

The analysis reveals distinct correlation shapes remain consistent across both model sizes and datasets, indicating that these patterns are fundamental characteristics of training paradigms rather than task-specific adaptations DOFA models exhibit a characteristic correlation shape across all datasets: highly organized feature correlations at small scales transitioning to near-independence at full scale TerraMind-1.0 models demonstrate identical correlation shapes across both base and large sizes, characterized by smooth and mostly monotonic decrease from moderate to low correlations Prithvi-EO-2. p. 8

Where the paper says it

0 (large) shows a saturating rather than monotonic structure. Feature correlation declines from moderate values (∼0.

0 models exhibit the distinctive non-monotonic U-shaped correlation pattern across both Prithvi-EO-2. p. 8
Where the paper says it

0 (600M) exhibits a qualitatively distinct structure across all three analyses. Feature correlation follows a non-monotonic trajectory across the compute range (Figure 4c), and the EVR slope shows a corresponding non-monotonic pattern (Figure 6).

0 (300M) and Prithvi-EO-2. p. 2
Where the paper says it

As shown in Table 1, widely used RS FMs span multiple orders of magnitude in parameters and computational budgets. For instance, Prithvi [19, 34] leverages Sentinel-2 and Landsat time-series data, while Satlas [1] introduces 300M+ annotations for supervision, and SSL4EO-S12 [35] provides vision transformer (ViT) encoders pretrained via contrastive objectives on a large-scale multi-seasonal Sentinel-2 dataset.

0 (600M) and all datasets

Conclusion

The findings establish post-hoc slimming as a practical deployment strategy for resource-constrained RS applications and as a diagnostic tool for representational redundancy in RS FMs The study confirms that distributed encoding across moderately-dimensional subspaces enables robust slimmability, while the most aggressive dimensionality concentration at low compute (DOFA) re19

References

How Much of a Model Do We Need? Redundancy and Slimmability in Remote Sensing Foundation Models

How Much of a Model Do We Need? Redundancy and Slimmability in Remote Sensing Foundation Models

How Much of a Model Do We Need? Redundancy and Slimmability in Remote Sensing Foundation Models

How Much of a Model Do We Need? Redundancy and Slimmability in Remote Sensing Foundation Models

Technical appendices and supplementary material

Feature Correlation at different Model Scales

TerraMind-1.0 (base) and large (Figure 15c and 15d)

Geobench: Toward foundation models for earth monitoring

How Much of a Model Do We Need? Redundancy and Slimmability in Remote Sensing Foundation Models

<ref:2601.22841#pg4>RS FMs @ 4 RS datasets (avg. p. 1

Where the paper says it

RS FMs @ 4 RS datasets (avg. ) CV DINOv2 @ ImageNet-10 CV MAE @ ImageNet-10 CV MAE @ ImageNet-100 RS FMs @ m-eurosat CV MAE @ m-eurosat 0.

)Figure 1a) Relative retention rate: RS FMs averaged across four RS classification datasets maintain over 85 % relative accuracy at 1 % compute budget, whereas CV DINOv2 and CV MAE retain on average only 30 % to 35 % on ImageNet-10 and a CV MAE only 12 % on ImageNet-100 at the same compute budget. p. 1
Where the paper says it

Legends formatted as: “model(s) @ evaluation data”. a) Relative retention rate: RS FMs averaged across four RS classification datasets maintain over 85 % relative accuracy at 1 % compute budget, whereas CV DINOv2 and CV MAE retain on average only 30 % to 35 % on ImageNet-10 and a CV MAE only 12 % on ImageNet-100 at the same compute budget.

Figure 2: Downstream performance with respect to relative compute requirements of eight pretrained RS FMs on:Figure 3: k-NN classification performance with respect to their relative compute requirements of a MAE-based and a MoCo-based trained model from scratch, each once with and once without slimmable training on: a) m-eurosat; and b) m-bigearthnet. p. 6
Where the paper says it

575 mAPµ Rel. Compute Requirement (b) Figure 3: k-NN classification performance with respect to their relative compute requirements of a MAE-based and a MoCo-based trained model from scratch, each once with and once without slimmable training on: a) m-eurosat; and b) m-bigearthnet.

Figure 4: Mean Feature Correlation of the test split of m-eurosat and m-bigearthnet from the geobench benchmark collection with respect to their relative compute requirements on pretrained RS FMs:Figure 10: k-NN and linear classification accuracy with respect to their relative compute requirements of:Figure 7: k-NN relative retention rate with respect to their relative compute requirements for slimmable only the FFN layer, only the MHSA layer, or both layers simultaneously of: a) DOFA (large); b) TerraMind-1. p. 7
Where the paper says it

Compute Requirement (b) 0. 40 (c) Figure 4: Mean Feature Correlation of the test split of m-eurosat and m-bigearthnet from the geobench benchmark collection with respect to their relative compute requirements on pretrained RS FMs: a) DOFA (large); b) TerraMind-1.

0 (large); and c) Prithvi-EO-2. p. 2
Where the paper says it

As shown in Table 1, widely used RS FMs span multiple orders of magnitude in parameters and computational budgets. For instance, Prithvi [19, 34] leverages Sentinel-2 and Landsat time-series data, while Satlas [1] introduces 300M+ annotations for supervision, and SSL4EO-S12 [35] provides vision transformer (ViT) encoders pretrained via contrastive objectives on a large-scale multi-seasonal Sentinel-2 dataset.

0 (600M) on m-eurosat; and d) DOFA (large); e) TerraMind-1. p. 5
Where the paper says it

Larger models within each architecture (DOFA (large), Prithvi-EO-2. 0 (600M), TerraMind-1.

0 (large); and f) Prithvi-EO-2. p. 2
Where the paper says it

As shown in Table 1, widely used RS FMs span multiple orders of magnitude in parameters and computational budgets. For instance, Prithvi [19, 34] leverages Sentinel-2 and Landsat time-series data, while Satlas [1] introduces 300M+ annotations for supervision, and SSL4EO-S12 [35] provides vision transformer (ViT) encoders pretrained via contrastive objectives on a large-scale multi-seasonal Sentinel-2 dataset.

0 (600M) on m-bigearthnet. p. 5
Where the paper says it

0 (300M) Prithvi-EO-2. 0 (600M) SSL4EO ViT-S (Dino) SSL4EO ViT-S (MoCo) TerraMind-1.

Figure 9: k-NN classification performance with respect to relative compute requirements of a MAEbased and a MoCo-based trained model from scratch, each once with and once without slimmable training on m-eurosat for different training durations:DOFA (base) Prithvi-EO-2. p. 6
Where the paper says it

575 mAPµ Rel. Compute Requirement (b) Figure 3: k-NN classification performance with respect to their relative compute requirements of a MAE-based and a MoCo-based trained model from scratch, each once with and once without slimmable training on: a) m-eurosat; and b) m-bigearthnet.

0 (300M) SSL4EO ViT-S (Dino) SSL4EO ViT-S (MoCo) TerraMind-1. p. 5
Where the paper says it

0 (300M) Prithvi-EO-2. 0 (600M) SSL4EO ViT-S (Dino) SSL4EO ViT-S (MoCo) TerraMind-1.

0 (base) TerraMind-1. p. 5
Where the paper says it

0 (600M) SSL4EO ViT-S (Dino) SSL4EO ViT-S (MoCo) TerraMind-1. 0 (base) TerraMind-1.

p. 5
Where the paper says it

0 (600M) SSL4EO ViT-S (Dino) SSL4EO ViT-S (MoCo) TerraMind-1. 0 (base) TerraMind-1.

0 (large)DOFA (base) DOFA (large) Prithvi-EO-2. p. 5
Where the paper says it

DOFA (base) DOFA (large) Prithvi-EO-2. 0 (300M) Prithvi-EO-2.

0 (300M) Prithvi-EO-2. p. 2
Where the paper says it

As shown in Table 1, widely used RS FMs span multiple orders of magnitude in parameters and computational budgets. For instance, Prithvi [19, 34] leverages Sentinel-2 and Landsat time-series data, while Satlas [1] introduces 300M+ annotations for supervision, and SSL4EO-S12 [35] provides vision transformer (ViT) encoders pretrained via contrastive objectives on a large-scale multi-seasonal Sentinel-2 dataset.

0 (600M) SSL4EO ViT-S (Dino) SSL4EO ViT-S (MoCo) TerraMind-1. p. 5
Where the paper says it

0 (300M) Prithvi-EO-2. 0 (600M) SSL4EO ViT-S (Dino) SSL4EO ViT-S (MoCo) TerraMind-1.

0 (base) TerraMind-1.0 (large)Figure 13: IoU with respect to relative compute requirements of eight pretrained RS FMs on m-SAcrop-type (10-class semantic segmentation). p. 5
Where the paper says it

Compute Requirement (d) Figure 2: Downstream performance with respect to relative compute requirements of eight pretrained RS FMs on: a) m-eurosat (single-labels scene classification); b) m-bigearthnet (multi-label scene classification); c) m-cashew-plant (semantic segmentation); d) OSCD (change detection). Results for m-brick-kiln, m-so2sat, and m-SA-crop-type are provided in Appendix G.

Figure 14: k-NN classification performance with respect to their relative compute requirements of a MAE-based and a MoCo-based trained model from scratch, each once with and once without slimmable training on: a) m-brick-kiln; and b) m-so2sat. p. 6
Where the paper says it

575 mAPµ Rel. Compute Requirement (b) Figure 3: k-NN classification performance with respect to their relative compute requirements of a MAE-based and a MoCo-based trained model from scratch, each once with and once without slimmable training on: a) m-eurosat; and b) m-bigearthnet.

Figure 15: Mean Feature Correlation of the test split of four classification datasets from the geobench benchmark collection with respect to their relative compute requirements on pretrained RS FMs:Figure 16: EVR with respect to the singular value rank index for eight RS FMs on four classification datasets from the geobench benchmark collection at three relative compute requirements:Figure 8: KNN classification performance with respect to relative compute requirements on the test split of m-eurosat when selecting the first, last, or a random subset of neurons for slimming. p. 7
Where the paper says it

Compute Requirement (b) 0. 40 (c) Figure 4: Mean Feature Correlation of the test split of m-eurosat and m-bigearthnet from the geobench benchmark collection with respect to their relative compute requirements on pretrained RS FMs: a) DOFA (large); b) TerraMind-1.

Figure 11: Effective Dimension of the test split of four classification datasets from the geobench benchmark collection with respect to their relative retention rate on pretrained RS FMs:Figure 6: Log-Log-Slopes of the EVR with relative compute requirements for three different RS FMs averaged across four RS classifications datasets. p. 7
Where the paper says it

00 Effective Dimension (b) 10 0 10 1 0. 00 (c) Figure 5: Effective Dimension of the test split of four classification datasets from the geobench benchmark collection with respect to their relative retention rate on pretrained RS FMs: a) DOFA (large); b) TerraMind-1.

Impact StatementL Licenses and AssetsK Computational Resources</ref:pg28

<ref:2601.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Post-hoc slimming of remote sensing foundation models can be used as a practical deployment strategy for resource-constrained applications because RS FMs retain 69 % to 109 % relative accuracy on RS datasets under aggressive width reduction.

  2. The effective dimensionality analysis reveals that task-relevant variance concentrates in few principal components and is redundantly encoded across model dimensions, suggesting that the most aggressive slimming preserves performance by removing this redundant information.

  3. Learned slimmable training improves MoCo-based models because it consistently outperforms regular training at low compute budgets while maintaining equivalent or superior performance at higher scales on m-eurosat.

  4. The analysis of feature correlation shows that distributed encoding across moderately-dimensional subspaces enables robust slimmability, indicating that the organization of information across scales is determined by the training paradigm rather than parameter count.

  5. Simultaneous slimming of both FFN and MHSA layers is recommended because it proves superior across all evaluated models, enabling compression to 1 % relative compute while exhibiting the most gradual performance degradation curves.

Abstract

Large-scale foundation models (FMs) in remote sensing (RS) (denoted as RS FMs) are developed following paradigms established in computer vision (CV), yet the validity of transferring CV scaling laws to RS has not been systematically examined. We hypothesize that RS FMs enter an overparameterized regime at substantially smaller scales than their CV counterparts, with task-relevant information encoded redundantly across model dimensions. To test this hypothesis, we apply post-hoc slimmability, uniform width reduction of pretrained encoder transformer blocks, to measure representational redundancy across ten state-of-the-art RS FMs on classification, segmentation, and change detection. RS FMs retain on average over 85% relative accuracy on RS classification datasets under aggressive width reduction, while masked autoencoder (MAE) and DINOv2 pretrained on natural images (denoted as CV MAE and CV DINOv2) degrade sharply on ImageNet subsets of matched class count over the same range of computational requirements. A CV MAE evaluated directly on the same RS dataset narrows but does not close the gap, indicating that both dataset characteristics and domain-specific pretraining contribute to the differences between the models. Analyses of the representation space of the models indicate that task-relevant variance concentrates in few principal components and is redundantly encoded across model dimensions. We further show that learned slimmable training improves over post-hoc slimmability for contrastive objectives, but reconstruction-based objectives do not benefit consistently from current slimmable training protocols. Our findings establish post-hoc slimming as a practical deployment strategy for resource-constrained RS applications and as a diagnostic tool for representational redundancy in RS FMs. Code is available at https://github.com/hackelle/slimmable-remote-sensing-foundation-models.

Sources

Related papers