Crop Mapping Evaluation under Sampling Bias
cs.LG, cs.CV
Submitted: 2026-10-31
Updated: 2026-10-31
Comments: 17 pages, 4 figures, 5 tables. Code: https://github.com/microsoft/PaPPI
Code: https://github.com/microsoft/pappi
License: http://creativecommons.org/licenses/by/4.0/
The gist: Government agencies use crop maps to estimate production, monitor food security, and set agricultural policy.
Terminology
Abstract
Government agencies use crop maps to estimate production, monitor food security, and set agricultural policy. These crop maps are usually validated against field labels, but these labels are rarely a random sample of the area the map covers. Field teams tend to label parcels near roads and skip ones that are small or hard to reach, so performance measured on held-out labels mainly describes the places they visited. We call the difference between this reported performance and performance over the whole mapped region the deployment performance gap. To estimate performance over the whole region, we use a weighted form of prediction-powered inference (PPI). The classifier's predictions on unlabeled points across the region give a first estimate, and its errors on the labeled points correct that estimate, with more weight on labels from under-sampled conditions. We test this in five European regions where crop registers label every declared parcel, so the true performance is known. Simulating six documented collection practices there produces gaps of up to 0.20 macro-F1. On the collection rule that creates the largest gap in each region, 5 km block cross-validation still overestimates macro-F1 by as much as 0.18, while PPI with weights estimated from satellite embeddings reduces the largest error across regions to 0.07. Importance-weighted cross-validation with the same weights does almost as well, so most of the improvement comes from the weights themselves. We also apply the method to three Kenyan datasets that have no region-wide reference labels. For a roadside dataset collected with helmet-mounted cameras, the estimated macro-F1 falls from 0.93 under random cross-validation to 0.61, although the correction relies on a small number of heavily weighted labels. Our code is available at https://github.com/microsoft/pappi.
Sources
- AlphaEarth Foundations: An embedding field model for accurate and efficient global mapping from sparse label data
- Prediction-Powered Inference with Inverse Probability Weighting
- Prediction-Powered Inference with Imputed Covariates and Nonuniform Sampling
- Spatially Robust Inference with Predicted and Missing at Random Labels
- Design-Based Prediction-Powered Inference for Spatial Data
- Beyond the Training Distribution: Evaluating Predictions Under Distribution Shift and Selection Bias
- Generalized Prediction-Powered Inference, with Application to Binary Classifier Evaluation
Related papers
- A Physics-Regulated Neural Framework for Learning 3D Grain Growth Dynamics
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis