BigEarthNet.txt: A Large-Scale Multi-Sensor Image-Text Dataset and Benchmark for Earth Observation
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: We're looking at the paper: "BigEarthNet.txt: A Large-Scale Multi-Sensor Image-Text Dataset and Benchmark for Earth Observation".
Jane: Here's the gist of the paper. Vision-language models (VLMs) have shown strong performance in computer vision (CV), yet their performance on remote sensing (RS) data remains limited due to the lack of large-scale, multi-sensor RS image-text datasets with diverse textual annotations <ref:2603.29630#pg2> The paper introduces BigEarthNet.txt, a largescale, multi-sensor image-text dataset designed to advance instructiondriven image-text learning in Earth observation across multiple tasks <ref:2603.29630#pg6>.
Tom: Dataset Composition and Scale BigEarthNet.txt contains 464 044 co-registered Sentinel-1 synthetic aperture radar and Sentinel-2 multispectral images with 9.6 M text annotations <ref:2603.29630#pg6> These annotations include geographically anchored captions describing landuse/land-cover (LULC) classes, their spatial relations, and environmental context <ref:2603.29630#pg6>, visual question answering pairs relevant for different tasks <ref:2603.29630#pg6>, and referring expression detection instructions for bounding box prediction <ref:2603.29630#pg6>.
Jane: The dataset supports 15 tasks (Presence, Area, Counting, Adjacency, Relative Position, Country, Season, and Climate Zone) across 4 broad categories <ref:2603.29630#pg6>.
Tom: Annotation Generation Pipeline The textual annotations for BigEarthNet.txt are generated through a threestage pipeline (template-based caption generation, large language model (LLM)-based linguistic augmentation, VQA and LULC annotation generation) <ref:2603.29630#pg6>.
Paper summary: Tom: So, we’ve covered the paper, BigEarthNet.txt: A Large-Scale Multi-Sensor Image-Text Dataset and Benchmark for Earth Observation. We looked at how they built it, what they found when they tested existing models against it, and what that means for future work in this area.
Jane: The authors are essentially saying that the main problem with current vision language models applied to remote sensing is the lack of large scale, multi-sensor datasets with diverse text annotations.
Lu: They claim BigEarthNet.txt solves this by providing a massive, co-registered dataset of Sentinel one synthetic aperture radar and Sentinel two multispectral images paired with nine point six million text annotations covering fifteen different tasks.
Meng: The implication for us is that if we can train models on this kind of data, they should be able to characterize the spatial and spectral details of these multi-sensor images much more accurately than what we're seeing now.
Tom: They show that fine tuning a model like InternVL3 one billion parameters adapted for multi-sensor input on BigEarthNet.txt leads to an improvement of thirty-one point five two percent on average compared to state of the art remote sensing models.
Jane: So, in simple terms, this work presents BigEarthNet.txt as a large scale multi-sensor RS image text dataset and benchmark split enabling reliable evaluation of VLM capabilities to model the complex spatial and spectral characteristics of multi-sensor RS data.
Lu: The paper is very promising for effective and interactive Earth observation because it enables natural language interactions with the complex content of RS images, thus democratizing the use of EO data for non experts as well.
Meng: If this dataset proves useful for real world applications, then we need to look into how to deploy models that can handle this complexity efficiently at scale.
Lalam: This is the paper's main contribution, and it sets a new standard for how we evaluate these kinds of AI systems when they deal with the intricacies of Earth observation imagery.
Conclusion: Tom: So we’re wrapping up our look at BigEarthNet.txt, the paper is titled "BigEarthNet.txt: A Large-Scale Multi-Sensor Image-Text Dataset and Benchmark for Earth Observation." It sounds like a lot of data and a big goal right there.
Jane: Yeah, it’s basically trying to solve that problem we talked about—how vision language models struggle with the complexity of satellite images—by creating this massive dataset. What are the authors actually aiming to achieve with this thing?
Lu: They're building a resource for instruction driven image text learning in Earth observation. They want to advance how these models understand multi-sensor data, which means they can do more than just look at one type of satellite image anymore.
Meng: From an engineering standpoint, I’m curious about the scale. How big is this dataset really? Because if you can't run it efficiently, it’s just a pretty picture on a screen for us.
Tom: They got some serious numbers there. We’re talking four hundred sixty-four thousand forty-four co-registered Sentinel one and Sentinel two images, plus nine point six million text annotations. That's massive scale we haven't seen in this space before.
Jane: And those annotations aren't just simple captions for what’s in the picture; they include things like spatial relations, land use classifications, and even instructions for detecting specific objects. It seems really thorough for training models to actually reason about the geography.
Lu: The annotation pipeline itself is interesting because it uses a three-stage process—template generation, then a large language model to augment the text, and finally some self-refinement—to make sure the text is rich without just adding junk information.
Meng: But what does this mean for deployment? If we fine tune a state of the art model on this, how much better are we actually getting in terms of real world accuracy versus just running a standard model on smaller data?
Tom: The results show that fine-tuning models like InternVL3 one billion parameters with multi-sensor input on BigEarthNet.txt gives them substantial performance gains across all fifteen tasks they defined. Specifically, the adapted remote sensing internVL model showed an improvement of thirty-one point five two percent on average compared to the best existing remote sensing models we looked at.
Jane: That thirty-one point five two percent gain is pretty concrete because it’s benchmarked against current state of the art RS models. It shows that this approach actually helps the AI understand those complex spatial and spectral characteristics better than before.
Lu: So, the big picture here is providing a reliable way to evaluate how well these vision language models can handle multi-sensor remote sensing data. It gives us a solid foundation to test new learning techniques for Earth observation tasks.
Meng: I see the value in having a benchmark split that’s manually verified and balanced across different land use classes too. That helps ensure the evaluation isn't just biased towards certain types of areas.
Tom: Exactly, it gives you the numbers you need to prove what works, which is crucial when we're talking about moving models from research labs into actual applications for remote sensing data.
Jane: And ultimately, this dataset is promising because it opens the door for natural language interactions with complex RS images, making Earth observation data accessible to people who aren't deep experts in the field. What do you all think this means for how we use satellite imagery in everyday stuff?
BIFOLD and Technische University of Berlin · University of Trento National Technical University and National Observatory of Athens
cs.CV
Submitted: 2026-03-31
Updated: 2026-11-10
Comments: Accepted at IEEE/CVF Winter Conference on Applications of Computer Vision 2027. Our code and data are publicly available at https://txt.bigearth.net
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 79/100
The gist: Vision-language models (VLMs) have shown strong performance in computer vision (CV), yet their performance on remote sensing (RS) data remains limited due to the lack of large-scale…
Key concepts
- BigEarthNet.txt
- This is a large, multi-sensor dataset containing 464,044 co-registered Sentinel-1 radar and Sentinel-2 multispectral images. It includes over 9.6 million text annotations describing land use/cover classes, their spatial relationships, and environmental context across various tasks.
- Annotation Generation Pipeline
- Text descriptions are created using a three-stage process: first, template-based generation extracts spatial attributes like class presence and adjacency; second, an LLM performs linguistic augmentation to diversify the text; finally, a self-refinement step corrects the output against the original template.
- Instruction-Driven Learning
- This refers to training vision models using natural language instructions rather than just labeled images. BigEarthNet.txt is designed to advance this by providing diverse textual annotations relevant for numerous tasks, allowing VLMs to learn how to interact with and interpret complex Earth observation data.
- Multi-Sensor Image-Text Dataset
- This dataset uniquely combines data from two different remote sensing sources: Sentinel-1 (synthetic aperture radar) and Sentinel-2 (multispectral imagery). This combination allows models to analyze both structural information from radar and spectral information from optical sensors simultaneously.
Terminology
Summary
Vision-language models (VLMs) have shown strong performance in computer vision (CV), yet their performance on remote sensing (RS) data remains limited due to the lack of large-scale, multi-sensor RS image-text datasets with diverse textual annotations <ref:2603. p. 0 net
Abstract. Vision-language models (VLMs) have shown strong performance in computer vision (CV), yet their performance on remote sensing
(RS) data remains limited due to the lack of large-scale, multi-sensor RS
image-text datasets with diverse textual annotations. To overcome this
bottleneck, recently proposed datasets shift toward large-scale, automatically
generated texts: RS5M [44] introduces the first million-scale corpus with 5 million image–caption pairs; GAIA [41] provides 205 150 pairs, with multi-sentence
captions emphasizing environmental dynamics; and Landsat30-AU [21] contains
196 262 image-caption pairs and 17 725 multiple-choice (MC) VQAs. For VQA
specifically, RSVQA [19] comprises 77 232 and 1 066 316 question-answer triplets
for low- and high-resolution imagery, while RSVQAxBEN [18] scales to approximately 15 million pairs with S2 BigEarthNet [31] RGB composites for CORINE
Land Cover (CLC) class presence. To address this limitation, we introduce BigEarthNet. txt, a largescale, multi-sensor image-text dataset designed to advance instructiondriven image-text learning in Earth observation across multiple tasks.Where the paper says it
Where the paper says it
Where the paper says it
Dataset Composition and Scale
BigEarthNet.txt contains 464 044 co-registered Sentinel-1 synthetic aperture radar and Sentinel-2 multispectral images with 9. p. 0 BigEarthNet. txt contains 464 044 co-registered Sentinel-1 synthetic
aperture radar and Sentinel-2 multispectral images with
9
. Keywords: Multi-Sensor Image-Text Dataset
· Vision-Language Models · Earth Observation · Remote Sensing
1 Introduction
Unprecedented advances in satellite technology have led to a significant increase
in the volume of Earth observation (EO) data archives (e. the Sentinel satellites
of the Copernicus program alone acquire roughly 20 TB of satellite images per
arXiv:2603. txt contains 464 044 co-registered Sentinel-1 synthetic
aperture radar and Sentinel-2 multispectral images with
9
. 6 M text annotations, including: i) geographically anchored captions describing landuse/land-cover (LULC) classes, their spatial relations, and environmental
context; ii) visual question answering pairs relevant for different tasks;
and iii) referring expression detection instructions for bounding box prediction. txt contains 464 044 co-registered Sentinel-1 synthetic
aperture radar and Sentinel-2 multispectral images with
9
. 6 M text annotations, including: i) geographically anchored captions describing landuse/land-cover (LULC) classes, their spatial relations, and environmental
context; ii) visual question answering pairs relevant for different tasks;
and iii) referring expression detection instructions for bounding box prediction. txt contains 464 044 co-registered Sentinel-1 synthetic
aperture radar and Sentinel-2 multispectral images with
9
. 6 M text annotations, including: i) geographically anchored captions describing landuse/land-cover (LULC) classes, their spatial relations, and environmental
context; ii) visual question answering pairs relevant for different tasks;
and iii) referring expression detection instructions for bounding box prediction. 6 million S1-
S2-text triplets. The dataset supports 15 tasks (Presence, Area, Counting, Adjacency,
Relative Position, Country, Season, and Climate Zone, denoted as Pr, A, Cnt, Adj, RP,
Loc, S, and Clt, respectively) across 4 broad categories.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Annotation Generation Pipeline
The textual annotations for BigEarthNet.txt are generated through a threestage pipeline (template-based caption generation, large language model (LLM)-based linguistic augmentation, VQA and LULC annotation generation) <ref:2603. p. 4 1 Annotation Generation Pipeline
The textual annotations for BigEarthNet. txt are generated through a threestage pipeline (template-based caption generation, large language model (LLM)-
based linguistic augmentation, VQA and LULC annotation generation). Template-based caption generation. We first construct captions by extracting
four categories of spatial attributes directly from the reference maps: the presence
of LULC classes, the count of individual contiguous regions per class (instances),
the size of each class in total and per instance, and pairwise spatial adjacency
between classes. First, a paraphrasing step diversifies lexical and syntactic structure while prohibiting the addition of unsupported information. Second, a self-refinement
step [22] revises the paraphrased output against the original template to eliminate hallucinated content and restore missing information.Where the paper says it
Where the paper says it
Where the paper says it
Benchmark Split and Evaluation
A manually verified benchmark split contains 1082 image pairs with manually verified text annotations for systematic evaluation of VLMs for EO tasks <ref:2603.29630#pg6>. Annotations are further balanced across LULC classes to the extent permitted by their natural distribution, and binary and MCQ annotations are balanced across answer options to expose answer-bias in evaluated models <ref:2603. p. 7 To expose answer-bias in evaluated models, binary and MCQ annotations are balanced across answer options. Annotations are further balanced across LULC classes to the extent permitted
by their natural distribution.Where the paper says it
Performance Findings
The results show that current VLMs in both CV and RS domains exhibit limited capability to characterize the complex spatial/spectral content of multi-sensor images <ref:2603.29630#pg5>. Fine-tuning a state-of-the-art (SOTA) VLM (InternVL3-1B [45]) adapted for multi-sensor input on BigEarthNet. p. 3 Using
our benchmark split, we demonstrate that current VLMs in both CV and RS
show limited ability to characterize the complex content of multi-sensor RS images. Fine-tuning a state-of-the-art (SOTA) VLM (InternVL3-1B [45]) adapted
for multi-sensor input (denoted as RS-InternVL) on BigEarthNet. Fine-tuning a state-of-the-art (SOTA) VLM (InternVL3-1B [45]) adapted
for multi-sensor input (denoted as RS-InternVL) on BigEarthNet. txt leads to
substantial performance gains across all considered tasks. Using
our benchmark split, we demonstrate that current VLMs in both CV and RS
show limited ability to characterize the complex content of multi-sensor RS images. Fine-tuning a state-of-the-art (SOTA) VLM (InternVL3-1B [45]) adapted
for multi-sensor input (denoted as RS-InternVL) on BigEarthNet. Keywords: Multi-Sensor Image-Text Dataset
· Vision-Language Models · Earth Observation · Remote Sensing
1 Introduction
Unprecedented advances in satellite technology have led to a significant increase
in the volume of Earth observation (EO) data archives (e. the Sentinel satellites
of the Copernicus program alone acquire roughly 20 TB of satellite images per
arXiv:2603.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Conclusion
In summary, the work presents BigEarthNet.txt as a large-scale multi-sensor RS imagetext dataset and benchmark split enabling reliable evaluation of VLM capabilities to model the complex spatial and spectral characteristics of multi-sensor RS data <ref:2603. p. 2 1, the existing RS imagetext datasets suffer from: i) limited availability of co-registered multi-sensor data
that includes more than three bands, and ii) limited diversity in text annotation
types. These limitations directly hinder the ability of VLMs to characterize: 1)
the complex spatial/spectral content of RS images; and also 2) complementary
information among multi-sensor data. 2 Related work
VLMs in RS enable natural language interactions with images across multiple tasks, including VQA, change captioning, and referring expression detection [14, 15, 30]. To facilitate the research and development of VLMs, several
image-text datasets have been proposed in RS.Where the paper says it
Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Fine-tuning a state-of-the-art (SOTA) VLM adapted for multi-sensor input, denoted as RS-InternVL, on BigEarthNet.txt leads to
substantial performance gains across all considered tasks.
This adaptation allows the model toeffectively query and interpret these vast EO archives
by jointly encoding visual observations and natural language for tasks like image captioning and VQA. -
Implement a three-stage annotation generation pipeline that includes template-based caption generation, LLM-based linguistic augmentation (paraphrasing and self-refinement), and specific VQA/referring expression detection annotations. This ensures
lexical diversity
andricher in the semantic content per caption,
as shown by the MTLD score surpassing existing datasets by more than 1.7x. -
Develop a benchmark split containing manually verified text annotations to ensure
reliable evaluation of VLM capabilities to model the complex spatial and spectral characteristics of multi-sensor RS data.
This systematic evaluation exposes limitations in current VLMs, showing thatRS-specific models lead to almost no advantage over their general-purpose counterparts
unless fine-tuned on rich data. -
Train a model like RS-InternVL on BigEarthNet.txt to achieve an
improvement of 31.52 % on average,
demonstrating thateven a small-scale VLM with simple adaptations can effectively model complex content of multi-sensor data when adequate training data is considered.
This makes complex LULC understanding accessible to non-experts by enablingnatural language interactions with the complex content of RS images.
Abstract
Vision-langugage models (VLMs) have shown strong performance in computer vision (CV), yet their performance on remote sensing (RS) data remains limited due to the lack of large-scale, multi-sensor RS image-text datasets with diverse textual annotations. Existing datasets predominantly include aerial Red-Green-Blue imagery, with short or weakly grounded captions, and provide limited diversity in annotation types. To address this limitation, we introduce BigEarthNet. txt, a large-scale, multi-sensor image-text dataset designed to advance instruction-driven image-text learning in Earth observation across multiple tasks. It contains 464044 co-registered Sentinel-1 synthetic aperture radar and Sentinel-2 multispectral images with 9.6M text annotations, including: i) geographically anchored captions describing land-use/land-cover (LULC) classes, their spatial relations, and environmental context; ii) visual question answering pairs relevant for different tasks; and iii) referring expression detection instructions for bounding box prediction. A comparative statistical analysis shows that BigEarthNet. txt surpasses existing RS image-text datasets in textual richness and annotation type variety. We further establish a manually-verified benchmark split and extensively evaluate existing VLMs in RS and CV. The results show the limitations of these models on tasks that involve complex LULC classes, whereas fine-tuning using BigEarthNet. txt results in consistent performance gains across all considered tasks. Our code and data are publicly available at https://txt.bigearth.net.
Sources
- Qwen3-VL Technical Report
- LLaVA-OneVision: Easy Visual Task Transfer
- Landsat30-AU: A Vision-Language Dataset for Australian Landsat Imagery
- EarthMind: Leveraging Cross-Sensor Data for Advanced Earth Observation Interpretation with a Unified Multimodal LLM
- DINOv3
- OpenAI GPT-5 System Card
- GLM-4.5V and GLM-4.1V-Thinking: Towards Versatile Multimodal Reasoning with Scalable Reinforcement Learning
- LLaMA: Open and Efficient Foundation Language Models
- BERTScore: Evaluating Text Generation with BERT
- InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models
Related papers
- TEA: Text Encoder Alignment for Robust Concept Erasure in Text-to-Image Models
- Do Egocentric Video-Language Models Capture Both Hand- and Object-Centric Cues?
- Aura: Consistent Multi-Subject Video Generation via VLM-Grounded Semantic Alignment
- PVRF: All-in-one Adverse Weather Removal via Prior-modulated and Velocity-constrained Rectified Flow
- DLWM: Diverse Latent World Models for Efficient Multimodal Reasoning
- Out of Sight, Still in Mind: Token Compression for Omni-LLMs