Contrastive Mask Fidelity: Reference-Free Auditing of Ground-Truth Masks in Remote Sensing Semantic Segmentation

The paper introduces Contrastive Mask Fidelity (CMF), a training-free, reference-free metric for auditing ground-truth masks in remote sensing semantic segmentation. The authors argue that "semantic segmentation models are trained and evaluated against human-drawn masks, yet remote-sensing annotations are often coarse, incomplete, or misaligned; high overlap scores may then reflect agreement with imperfect labels rather than faithfulness to the image—an evaluation paradox. CMF addresses this by scoring competing class masks directly against image evidence: CMF composites keep and erase counterfactual views of each mask and asks a frozen vision-language judge whether class evidence is concentrated inside the mask and absent outside."

The method is part of a three-stage auditing system called GeoVeritas. Stage I uses Seg-Probe, a training-free open-vocabulary probe built on SegEarth-OV3, to generate candidate masks. Seg-Probe matches or exceeds SegEarth-OV3 on nine of ten datasets and achieves the best average performance (56.3 vs. 55.4) among nine training-free methods. Stage II applies CMF, which uses a frozen RemoteCLIP judge to score keep and erase views of both the human annotation and the prediction, computing a fidelity score F(Mc) = P(cxkc) − P(cxec). A prediction is judged more faithful when ∆F = F(Mcp) − F(Mca) > 0. Stage III arbitrates labels per class, substituting the prediction only when ∆F > δ, with ties deferring to the human label.

The audit covers 10,731 image-class pairs across ten remote-sensing benchmarks spanning satellite, airborne, and UAV platforms with GSD from 0.05 to 1 m. Results show predictions are preferred on seven of ten datasets, with a pooled win rate of 56.6% and a dataset-macro average of 57.0%. The verdicts exhibit "a consistent class structure across resolutions and platforms: well-defined objects such as buildings, cars, and roads tend to favor predictions, whereas amorphous land-cover classes such as cropland, agriculture, and urban areas more often favor human annotations. For example, cars reach 84.6% and buildings 75.2% on Vaihingen, buildings 76.6–77.9% on LandCover.ai and FloodNet, and roads 76.2% on Massachusetts Roads, whereas LoveDA agriculture falls to 35.4% and DeepGlobe urban land to 23.5%."

Validation against expert judgment used 300 image–class pairs from six representative datasets with three annotators; after excluding 21 split votes, CMF achieves 81.0% agreement with the consensus, outperforming simpler adjudication rules by at least 10.3 percentage points, with bootstrap intervals [76.3, 85.3] against [65.4, 75.8] for the strongest baseline. Agreement reaches 89.3% on man-made classes and 88.4% for high-margin verdicts.

Cross-domain transfer experiments trained a DeepLabV3+ segmenter under five supervisions: raw annotation, Seg-Probe only, confidence replacement, random replacement, and CMF replacement. Results show "a consistent ordering on all four directions: CMF > confidence > Seg-Probe-only > raw GT, with random replacement statistically indistinguishable from raw GT. CMF beats Seg-Probe-only by +2.1 mIoU on average, beats confidence replacement by +1.2 at matched coverage, and random replacement yields no benefit (+0.3, within one std). Gains are concentrated on man-made classes: the CMF-trained model closes building footprints and reconnects road segments that the GT-trained model reproduces only partially, while amorphous land cover—the classes on which CMF usually defers to the annotation—is reproduced comparably by both models."

Sensitivity analyses show the contrastive design reduces the correlation with mask area from ρ=0.61 to ρ=0.08. Validity controls detect translation on 70–92% of pairs and erosion on 52–86%, but are less sensitive to dilation (0–61%). Judge swaps to CLIP, OpenCLIP, and GeoRSCLIP all reproduce the man-made vs. amorphous class structure (ρ≥0.8 agreement between per-class win-rate profiles). Compositing choices are stable: Gaussian-blur fill outperforms black and mean-color fills (shift detection 0.85 vs. 0.71/0.79), and verdicts vary by at most 0.6 points across τ ∈ 50, 100, 200.

The paper acknowledges limitations: CMF inherits the biases of its frozen judge and measures class-relative semantic fidelity under a specified vocabulary rather than precise boundary geometry or instance topology, and the audit is label-conditioned: it excludes classes that only the prediction claims. The expert study covers only 300 pairs from six datasets. The authors position CMF as a scalable screening tool, not a replacement for expert annotation: it surfaces disagreements that overlap metrics cannot resolve and prioritizes masks for human review.

Improvements to AI systems:

1. Add a training-free, reference-free mask-auditing layer to any semantic segmentation pipeline. The improved system can evaluate whether a predicted or human-labeled mask truly aligns with image evidence, independent of ground-truth quality. It outputs a per-class fidelity score and a binary verdict (prediction vs. annotation) for each image–class pair, flagging low-fidelity masks for human review without retraining.

2. Implement a three-stage GeoVeritas-style audit loop for open-vocabulary segmentation. The system first generates candidate masks via a training-free open-vocabulary probe (e.g., SegEarth-OV3), then applies Contrastive Mask Fidelity (CMF) with a frozen vision-language judge (e.g., RemoteCLIP) to score keep/erase counterfactual views, and finally arbitrates per class—substituting the prediction only when the fidelity margin exceeds a tunable threshold δ, otherwise deferring to the human label. This yields a corrected training set with class-specific, confidence-aware mask replacements.

3. Enable cross-domain label correction for downstream model training. The improved system can retrain a segmenter (e.g., DeepLabV3+) on a dataset where masks have been audited and selectively replaced by CMF. It consistently outperforms training on raw annotations, Seg-Probe-only masks, or confidence-based replacement, with gains concentrated on man-made classes (buildings, roads, cars) while preserving performance on amorphous land cover (cropland, urban areas). The system automatically decides which masks to replace and which to keep, based on the fidelity margin.

4. Provide a class- and resolution-aware audit report. The system can generate per-class win-rate statistics across platforms (satellite, airborne, UAV) and ground sample distances (0.05–1 m). It identifies that well-defined objects (cars, buildings, roads) favor predictions, while amorphous classes (agriculture, urban land) favor human annotations. This allows users to prioritize human review for classes where the judge is less reliable or where annotations are systematically preferred.

5. Add a validity-control mechanism to detect mask misalignments. The system can automatically test for translation and erosion artifacts in masks (detecting 70–92% of translations and 52–86% of erosions) by applying controlled perturbations and checking whether the fidelity score drops. This helps flag annotation errors before training, reducing noise propagation.

6. Enable judge-agnostic auditing with robustness guarantees. The system can swap the frozen vision-language judge (e.g., RemoteCLIP, CLIP, OpenCLIP, GeoRSCLIP) and still reproduce the same man-made vs. amorphous class structure (ρ≥0.8 agreement between per-class win-rate profiles). This makes the audit portable across different backbone models without recalibration.

7. Offer a compositing-parameter-insensitive scoring pipeline. The system uses Gaussian-blur fill for counterfactual views (outperforming black or mean-color fills in shift detection) and remains stable across blur radii τ ∈ 50, 100, 200 (verdict variation ≤0.6 points). This ensures reproducible audits without hyperparameter tuning.

8. Produce a prioritized review queue for human annotators. Given a large dataset, the system ranks image–class pairs by ΔF (absolute fidelity margin) and by class type (e.g., high-margin man-made classes first). In validation, high-margin verdicts achieve 88.4% expert agreement, so the system can surface only the most confident disagreements for efficient human arbitration, reducing annotation workload.

9. Support label-conditioned auditing with explicit uncertainty. The system reports that it excludes classes only the prediction claims (label-conditioned audit). It can output a flag when a prediction proposes a class absent from the annotation, prompting a separate out-of-vocabulary review—useful for discovering missing or novel classes in remote-sensing imagery.

10. Provide a bias-aware confidence score. Since CMF inherits judge biases and measures class-relative semantic fidelity (not boundary geometry), the system can output a per-class reliability index (e.g., high for cars/buildings, low for cropland/urban) and a warning when the judge’s preference deviates from expert consensus. This allows downstream users to discount verdicts on amorphous classes or to switch judges for those classes.

Semantic segmentation models are trained and evaluated against human-drawn masks, yet remote-sensing annotations are often coarse, incomplete, or misaligned; high overlap scores may then reflect agreement with imperfect labels rather than faithfulness to the image, creating an evaluation paradox. We introduce Contrastive Mask Fidelity (CMF), a training-free, reference-free metric that scores competing class masks directly against image evidence. CMF composites keep and erase counterfactual views of each mask and asks a frozen vision-language judge whether class evidence is concentrated inside the mask and absent outside. We validate CMF on controlled mask corruptions, then audit 10,731 image-class pairs across ten remote-sensing benchmarks using candidate masks from Seg-Probe, a training-free open-vocabulary probe built on SegEarth-OV3 that outperforms prior baselines on nine of ten datasets. The audit reveals systematic, class-dependent annotation distortion: man-made classes such as buildings, roads, and cars favor the candidate mask on 62-85% of pairs, whereas ambiguous land cover more often favors human annotations. On a blinded three-annotator consensus, CMF matches expert judgment on 81% of pairs, exceeding keep-only scoring, model confidence, and a trained label-quality baseline. Finally, conservative class-wise arbitration yields supervision that improves cross-domain transfer over raw annotations and matched replacement controls, positioning CMF as a scalable tool for auditing ground truth rather than presuming it infallible.