Filling Before Advancing: Capability-Gap-Driven Post-Training for Scenario-Specialized Remote Sensing MLLMs

arXiv:2607.22205 · cs.CV, cs.AI · Submitted 2026-07-24 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "Filling Before Advancing".

Jane: The gist Remote sensing multimodal large language models (RSMLLMs) require fine-grained scenario specialization, constrained by scarce high-quality scenario data and incomplete capability coverage,

Tom: First, who's behind it and why it matters.

Paper summary: Tom: So we're looking at this paper, "Filling Before Advancing: Capability-Gap-Driven Post-Training for Scenario-Specialized Remote Sensing MLLMs." The core idea here is that instead of just trying to jump straight to a specific scenario, like analyzing a harbor, we need to fill in some foundational gaps first.

Jane: That makes sense. Because right now with remote sensing multimodal large language models, they can understand general aerial images pretty well, but when you need them for something specific—say, understanding a coastal harbor—you run into two big problems: not enough really high-quality scenario data and the models just don't have all the necessary capabilities yet.

Lu: Exactly. The paper argues that single-stage supervised fine-tuning just doesn't cut it because you end up trying to fit too much adaptation into those limited high-quality samples, which is tough for both things simultaneously.

Meng: I worry about that data scarcity thing though, Lu. If we try to give the model a massive amount of specific harbor data right away, it might just overfit and not generalize well to new types of harbors or even different conditions.

Lalam: The paper suggests a post-training route called FBA, which means filling those capability gaps sequentially before you advance toward scenario specialization >

Tom: Right, so this FBA thing is basically about closing off these multi-level capability gaps in order before you start tailoring the model for a specific task. It lays out three stages for that process.

Jane: That sounds like a structured way to approach the learning problem, which is helpful when you don't have all the perfect data right from the start. What are those first steps they suggest?

Lu: They propose RS semantic anchoring as stage one, which is focused on getting that broad overhead-view visual-language alignment down first >

Tom: Stage one sets the foundation for understanding general aerial imagery and language in this domain before we even think about harbor specifics. It’s establishing that initial baseline capability.

Meng: Okay, so after that, what's stage two? Because I’m thinking about how much compute we need to bridge those general priors to the specific needs of a harbor.

Lalam: Stage two is called domain-bridge convergence, which learns those shared remote sensing priors across different scenarios using multiple modalities >

Paper summary: Jane: So it’s about making sure the model understands things like sensor awareness and bridging context, even when switching between different types of imagery. That sounds like a way to share knowledge more effectively.

Tom: And then stage three is evidence-grounded scenario tuning, which is where you finally focus on the harbor-specific downstream tasks using that accumulated knowledge >

Meng: I’m wondering about the data progression they use for this, since we know high-quality scenario data is scarce. How do they handle that scarcity across those three stages?

Lu: They use something called CPRS, which is a three-layer supervision dataset comprising RS-Anchor, Bridge-Conv, and ScenarioEG >

Jane: And the way they structure that CPRS dataset is interesting because the proportion of harbor-related remote sensing images increases progressively through those layers. It goes from fifteen percent in D1 to fifty-five percent in D2 and eighty-five percent in D3 >

Tom: So stage one, RS-Anchor, uses existing RGB remote sensing captioning to get that initial broad alignment >

Jane: That’s building on what's already out there from things like RGB RS captioning to establish that general overhead view understanding first. It’s using what we already have.

Meng: Then stage three, ScenarioEG, focuses heavily on those harbor-specific downstream tasks and strengthening responses under negative or ambiguous conditions >

Lalam: And the final pool of supervision they use for this tuning stage is selected from a larger intermediate set and audited to remove things like malformed conversations or missing image references >

Tom: So it’s a layered approach where you progressively specialize, using the output of the previous layer as input for the next one. It’s about sequential capability building rather than one big shot.

Jane: That sequential nature really emphasizes that we aren't just dumping everything in at once; we're carefully layering the complexity. This makes sense if you think about how a model needs to learn different kinds of skills, like general vision versus specific spatial reasoning.

Lu: The paper also introduces HarborEval, which is an eight-track diagnostic benchmark designed to test perception, spatial understanding, robustness, and generation >

Tom: That benchmark seems important because it gives us a comprehensive way to check performance beyond just one simple accuracy number. It covers a lot of ground for the final specialized model.

Jane: And the results they show are pretty compelling when you compare their FBA method against direct SFT and collapsed training methods on models like LLaVA-v1 point 5 and Qwen3-VL > <ref:2607.22205#pg1>

Paper summary: Meng: I’m looking at those scores, it shows FBA raising the HarborEval score from fifty-seven point ninety five to seventy point twenty nine for LLaVA-v1 point 5 when using the same final task supervision pool > <ref:2607.22205#pg1>

Tom: That's a significant jump compared to direct SFT, which got fifty-seven point ninety five as well, and it’s also better than collapsed training methods on Qwen3-VL where they went from eighty-one point zero nine to eighty-three point thirty seven >

Lalam: They also outperformed collapsed training and led the comparisons on specific subsets like harbor-related VRSBench and RSVQA, as well as expert scores from OpenEval >

Jane: So what this means for someone who just listens, it suggests that if you have limited high-quality supervision, building those capabilities in stages is a more effective strategy than trying to do everything at once >

Tom: And the role-replacement analyses they did are telling us exactly which part of that process gives the most value; RS-Anchor and ScenarioEG both show consistent gains >

Meng: The bridging transfer analysis is also interesting because it shows that the bridging centroid achieves a cosine similarity of zero point nine two to the harbor centroid, which is higher than general remote sensing or natural image samples >

Lu: That suggests the domain-bridge convergence step is successfully learning priors that are relevant to the target scenario, not just random connections between modalities >

Jane: So if we boil it down for a listener who doesn't read papers, this paper means that when you need an AI to get really specialized at something complex like analyzing a harbor with satellite data, you shouldn't just feed it all the specific examples and hope for the best >

Tom: You have to build up the general remote sensing understanding first through those staged steps before you try to tackle that hard scenario task >

Meng: From a deployment standpoint, this ordered approach seems smarter than just collapsing everything into one big training run because it keeps things modular and gives us checkpoints.

Lalam: The overall implication is that FBA provides a more reliable path for achieving scenario specialization when the necessary high-quality data isn't abundant >

Jane: So in short, this paper proposes a structured way to adapt remote sensing models for specific uses by filling capability gaps systematically before specializing them on the target scenario.

Conclusion: Tom: So we’ve gone through the setup for this paper, "Filling Before Advancing." Basically, they’re tackling that problem where remote sensing models can do general stuff but struggle when you want them to handle a very specific task, like analyzing a harbor.

Jane: Right. And instead of just throwing more data at it hoping it learns the specific thing, they have this idea called FBA—Filling Before Advancing. It’s about making sure the model fills in its general knowledge gaps before it tries to master that niche scenario >

Lu: Exactly, and they structure that filling into three ordered stages: semantic anchoring for general understanding, then domain bridging across different sensors, and finally evidence-grounded tuning for the specific harbor task >

Meng: So what’s the big picture here? What does this actually mean when we think about deploying these models in the real world? Is this just a neat training trick or does it actually solve a practical deployment problem >

Tom: It’s about making sure that when you take one of these remote sensing models and adapt it for something like coastal harbor understanding, you aren't starting from zero capability. You have to build up the prerequisites first >

Jane: And the results they show are pretty strong when they compare this FBA route against just doing direct fine-tuning or collapsing all that supervision into one big step >

Lu: The numbers on their HarborEval benchmark are what really stand out. They show a clear upward trend in performance across those stages, and the final score is quite high compared to the alternatives >

Tom: I’m looking at LLaVA-v1 point five and Qwen3-VL results, where FBA managed to raise those scores significantly over direct SFT by focusing on that ordered process >

Jane: So for someone just listening, this means when you want a specialized AI for remote sensing, the best way isn't to just give it all the harbor pictures right away but to systematically build its foundational skills first >

Meng: That makes sense from an engineering standpoint. It’s less risky to train in stages than trying to find one massive dataset that covers everything perfectly >

Lu: And they even used a lightweight urban instantiation, which shows this staged approach works well across different scenarios, not just for harbors >

Tom: It suggests that capability-driven training is a more robust way to handle the scarcity of high-quality scenario data in remote sensing AI >

Jane: So, the core idea here is that careful scaffolding of abilities leads to better final performance on specialized tasks. Next up, we’re going to look at how they actually built those three stages of supervision and what kind of data went into them.

Institute of Robotics and Automatic Information System (IRAIS) · College of Artificial Intelligence, Nankai University · Hyperspectral Computing Laboratory, Department of Technology of Computers and Communications, Escuela Politécnica, University of Extremadura · Faculty of Electrical and Computer Engineering, University of Iceland

cs.CV, cs.AI

Submitted: 2026-07-24

Updated: 2026-10-15

Code: https://github.com/Z0ngL1ng/filling-before-advancing

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 91/100

The gist: The gist Remote sensing multimodal large language models (RSMLLMs) require fine-grained scenario specialization, constrained by scarce high-quality scenario data and incomplete capability coverage,

Key concepts

Filling Before Advancing (FBA): A capability-gap-driven post-training strategy. Instead of trying to specialize immediately, FBA first develops necessary prerequisite skills in a structured sequence. This involves three ordered stages: establishing a general semantic basis, converging shared knowledge across different data types, and finally tuning the model specifically for the target scenario.
CPRS Dataset: A progressive dataset designed for coastal harbor understanding. It is built in three layers (RS-Anchor, Bridge-Conv, ScenarioEG), where the proportion of harbor-related imagery increases from 15% to 85%. This structure ensures the model learns broad remote sensing knowledge first before focusing intensely on specific harbor details.
HarborEval Benchmark: An eight-track diagnostic benchmark used to rigorously assess a model's performance in understanding harbors. It tests multiple aspects including perception, spatial reasoning, robustness against ambiguity, and generation capabilities. This allows for a comprehensive comparison of different training methods.

Terminology

Adaptation — 1× in this paper · explained in 1 episode
Alignment — 1× in this paper
Benchmark — 1× in this paper · explained in 2 episodes
Coastal-Port Remote Sensing — 1× in this paper
Fine-Tuning — 1× in this paper · explained in 5 episodes
Robustness — 1× in this paper · explained in 5 episodes
Supervised Fine-Tuning (SFT) — 1× in this paper · explained in 8 episodes

Summary

The gist Remote sensing multimodal large language models (RSMLLMs) require fine-grained scenario specialization, constrained by scarce high-quality scenario data and incomplete capability coverage, leading to the proposed filling before advancing (FBA) paradigm. <ref:2607.22205#pg2>

Motivation and Problem Statement

The paper addresses the gap between universal capability and scenario-specialized usability of RS-MLLMs, where traditional single-stage supervised fine-tuning (SFT) is insufficient due to two main challenges: scarce high-quality scenario data and difficult capability coverage <ref:2607.22205#pg2> The discrepancy discloses a gap between universal capability and scenario-specialized usability of RS-MLLMs, as illustrated in Fig. 1 (c) <ref:2607.22205#pg2> The point is less to add data than to assign scarce and costly supervision to separable roles <ref:2607.22205#pg2>

The FBA Paradigm

FBA is formulated as a capability-gap-driven post-training problem, where filling refers to sequentially closing multi-level capability gaps before advancing toward scenario specialization <ref:2607.22205#pg2> This involves three ordered stages: (1) RS semantic anchoring for overhead-view visual-language alignment; (2) domain-bridge convergence for shared RS priors across target and bridging scenarios under different modalities; and (3) evidencegrounded scenario tuning for downstream performance <ref:2607.22205#pg2>

Progressive Data Curation (CPRS)

The instantiation of FBA uses CPRS, a three-layer supervision dataset comprising RS-Anchor, Bridge-Conv, and ScenarioEG <ref:2607.22205#pg2> The proportion of harbor-related RS images increases progressively from 15% in D1 to 55% in D2 and 85% in D3 <ref:2607.22205#pg2> Stage 1, RS-Anchor, is constructed from existing RGB RS captioning (Lu et al. 2018; Cheng et al. 2022; Ge et al. 2025) to establish broad overhead-view visual-language alignment <ref:2607.22205#pg2> Stage 3, Scenario-EG, focuses on harbor-specific downstream tasks and strengthens evidence-grounded responses under negative or ambiguous conditions <ref:2607.22205#pg2>

Convergent Staged Training

The model adaptation is formulated as Mk = Adapt (Mk−1, Dk), k ∈ 1, 2, 3 <ref:2607.22205#pg2> Stage S1 adapts the initial general-purpose MLLM M0 using RS-Anchor supervision D1 to establish a broad RS semantic basis <ref:2607.22205#pg2> Stage S3 applies evidence-grounded scenario tuning to M2 using Scenario-EG supervision D3, building on the capabilities established in the previous stages <ref:2607.22205#pg2>

Evaluation and Results

An eight-track diagnostic benchmark called HarborEval is constructed to cover perception, spatial understanding, robustness, and generation <ref:2607.22205#pg2> Under comparable training budgets, FBA consistently outperforms Direct-SFT and Collapsed-SFT with both LLaVA-v1.5 and Qwen3-VL backbones <ref:2607.22205#pg2> For example, on HarborEval for LLaVA-v1.5, FBA achieves an overall score of 70.29, substantially outperforming Direct-SFT (57.95) <ref:2607.22205#pg2> The results show that the ordered post-training route is more effective than either direct target-scenario tuning or collapsing bridge-domain and scenario supervision into a single stage <ref:2607.22205#pg2>

Capability Role Validation

Role-replacement controls verify the intended capability role of each stage, showing that RS-Anchor consistently outperforms generic image-text supervision across both intermediate and downstream evaluations <ref:2607.22205#pg2> Bridge-Conv improves all six metrics over non-bridging supervision, supporting the use of target-related bridging scenes for multisource harbor adaptation <ref:2607.22205#pg2> Scenario-EG delivers the largest gains on HarborEval and OpenEval while preserving the previously established intermediate capabilities <ref:2607.22205#pg2> Bridging Transfer Analysis shows that the bridging centroid achieves a cosine similarity of 0.92 to the harbor centroid, exceeding that of general RS (0.73) and natural-image samples (0.54) <ref:2607.22205#pg2>

Conclusion

FBA achieves superior performance across comprehensive harbor diagnostics benchmarks over direct and collapsed alternatives on LLaVA-v1.5 and Qwen3-VL backbones <ref:2607.22205#pg2> The harbor instantiation serves as a practical example for scenario-specialized RSMLLMs under limited high-quality supervision <ref:2607.22205#pg2>

Acknowledgments

This work is supported by the National Natural Science Foundation of China under grants 62571271 and 62201552; the Natural Science Foundation of Tianjin under Grant No. 24JCQNJC01890, and the Fundamental Research Funds for the Central Universities <ref:2607.22205#pg2>

Supplementary Material Details

The construction of each supervision layer is detailed below, with Stage 1 constructing RS-Anchor from existing RGB RS captioning (Lu et al. 2018; Cheng et al. 2022; Ge et al. 2025) <ref:2607.22205#pg2> Stage 3 constructs Scenario-EG, comprising 53,000 harbor-specialized instruction samples that further concentrate supervision on evidence-grounded behavior in the target scenario <ref:2607.22205#pg2> The final pool is selected from a larger intermediate set and audited to remove malformed conversations, missing image references, metadata and task-field leakage <ref:2607.22205#pg2> HarborEval is constructed as a scenario-diagnostic evaluation set rather than as another training corpus <ref:2607.22205#pg2> The headline score macro-averages all eight diagnostic tracks after their normalization to a common 0–100 scale <ref:2607.22205#pg2> The final HarborEval score Heval is computed as the unweighted average of all eight diagnostic tracks after their normalization to a common 0–100 scale <ref:2607.22205#pg2> The overall HarborEval score Heval is computed as the unweighted average of all eight diagnostic tracks after their normalization to a common 0–100 scale <ref:2607.22205#pg2> The final HarborEval score Heval is computed as the unweighted average of all eight diagnostic tracks after their normalization to a common 0–100 scale <ref:2607.22205#pg2> The final HarborEval score Heval is computed as the unweighted average of all eight diagnostic tracks after their normalization to a common 0–100 scale <ref:2607.22205#pg2> The final HarborEval score Heval is computed as the unweighted average of all eight diagnostic tracks after their normalization to a common 0–100 scale <ref:2607.22205#pg2> The final HarborEval score Heval is computed as the unweighted average of all eight diagnostic tracks after their normalization to a common 0–100 scale <ref:2607.22205#pg2> The final HarborEval score Heval is computed as the unweighted average of all eight diagnostic tracks after their normalization to a common 0–100 scale <ref:2607.22205#pg2> The final HarborEval score Heval is computed as the unweighted average of all eight diagnostic tracks after their normalization to a common 0–100 scale <ref:2607.22205#pg2> The final HarborEval score Heval is computed as the unweighted average of all eight diagnostic tracks after their normalization to a common 0–100 scale <ref:2607.22205#pg2> The final HarborEval score Heval is computed as the unweighted average of all eight diagnostic tracks after their normalization to a common 0–100 scale <ref:2607.22205#pg2> The final HarborEval score Heval is computed as the unweighted average of all eight diagnostic tracks after their normalization to a common 0–100 scale <ref:2607.

Improvements for AI systems

Improve model adaptation by shifting from single-stage supervised fine-tuning (SFT) over target-domain samples to a capability-gap-driven post-training route called FBA, which first fills prerequisite capability gaps before advancing toward scenario specialization. This allows the system to progressively close multi-level capability gaps, encompassing overhead-view visual semantics, sensor-aware observability, target-bridging context, evidence grounding, and calibrated rejection.
Implement a three-stage progressive adaptation pipeline: (1) RS semantic anchoring using RSAnchor supervision to establish broad overhead view visual-language alignment; (2) domain-bridge convergence using Bridge-Conv supervision to learn RS priors shared across target and bridging scenarios under different modalities; and (3) evidencegrounded scenario tuning using Scenario-EG supervision for harbor-specific downstream tasks.
Construct an eight-track diagnostic benchmark called HarborEval, which assesses perception, spatial understanding, robustness, and generation by probing specific capability roles like T1 Object/scene, T5 Observability, and T8 Non-harbor rejection. This allows for a comprehensive diagnosis of the specialized model's performance across various critical dimensions.
Utilize the CPRS supervision dataset (RS-Anchor, Bridge-Conv, Scenario-EG) to curate data progressively: D1 focuses on broad RGB remote-sensing visual-language alignment, D2 extends supervision to target harbor scenes and coastal-port bridging scenes across RGB, SAR, PAN, and NIR, and D3 concentrates on harbor-specialized instruction samples for final tuning.
Employ staged multi-teacher (SMT) distillation during Stage 2 construction: metadata grounding normalizes source and modality tags under sensor-specific observability constraints, instruction synthesis then produces ShareGPT-style image–instruction–answer records, and a strict verifier determines if the sample is kept, rewritten, or dropped. This mechanism controls hallucination risks introduced by heterogeneous sources and sensor-dependent observability.
Ensure the final model achieves superior overall performance by comparing FBA against alternatives: Direct-SFT (57.95) and Collapsed-SFT, demonstrating that the ordered post-training route is more effective than either direct target-scenario tuning or collapsing bridge-domain and scenario supervision into a single stage.

Abstract

Remote sensing multimodal large language models (RS-MLLMs) have advanced aerial-image understanding, but scenario specialization remains constrained by scarce high-quality supervision and incomplete capability coverage. We propose filling before advancing (FBA), a capability-gap-driven post-training route that develops prerequisite capabilities before scenario-specific adaptation. For coastal harbor understanding, we construct CPRS (Coastal-Port Remote Sensing), a three-layer dataset covering RGB, SAR, PAN, and NIR imagery. FBA organizes supervision into RS semantic anchoring, multi-source bridging, and evidence-grounded scenario tuning. We also introduce HarborEval, an eight-track benchmark assessing perception, spatial understanding, robustness, and generation. Using the same final-task supervision pool, FBA raises the HarborEval score from 57.95 to 70.29 on LLaVA-v1.5 and from 81.09 to 83.37 on Qwen3-VL compared with Direct-SFT. It also outperforms collapsed training and leads the evaluated comparisons on harbor-related VRSBench and RSVQA subsets and expert-scored OpenEval. Stage-wise and supervision-replacement analyses examine the contributions of the supervision roles. A lightweight urban instantiation further supports cross-scenario applicability, with the proposed stage order achieving stronger overall performance than the tested alternatives. Project resources and release updates are available at https://github.com/Z0ngL1ng/filling-before-advancing.