Towards LLM Agents for Earth Observation
cs.AI
Submitted: 2025-04-16
Updated: 2026-10-05
Comments: Accepted at ACL 2026 Findings and ICML 2025 Workshop TerraBytes. Project page: https://iandrover.github.io/2025_univearth/
Project page: https://iandrover.github.io/2025_univearth
License: http://creativecommons.org/licenses/by-nc-nd/4.0/
The gist: Earth Observation (EO) provides critical planetary data for environmental monitoring, disaster management, climate science, and other scientific domains.
Terminology
Abstract
Earth Observation (EO) provides critical planetary data for environmental monitoring, disaster management, climate science, and other scientific domains. In this work we ask: Are AI systems ready for reliable Earth Observation? To answer this, we introduce UnivEARTH, a coding benchmark of 408 yes/no questions from NASA Earth Observatory articles across 7 various topics and over 15 satellite instruments and sources. Using Google Earth Engine API as a tool in a zero-shot setup, LLM agents achieve an accuracy of 40.0% where the code fails to run over 44% of the time. To better understand LLM agent behavior, we also analyze the impact of using the JavaScript API versus Python and the effect of providing documentation. Furthermore, we find that using a Reflexion framework significantly reduces errors: Claude-4.5-Sonnet, Gemini-2.5-Pro, and GPT-5 accuracies rise to around 60%. However, these results remain only marginally above random chance. Taken together, our findings identify significant challenges to be solved before AI agents can automate earth observation, and suggest paths forward.
Sources
- Probing the limitations of multimodal language models for chemistry and materials research
- MDCrow: Automating Molecular Dynamics Workflows with Large Language Models
- Auto-Discovery-Bench: Diagnosing Structured State Tracking in Oracle-Guided Discovery
- GEOBench-VLM: Benchmarking Vision-Language Models for Geospatial Tasks
- DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
- SciKnowEval: Evaluating Multi-level Scientific Knowledge of Large Language Models
- Earth-Agent: Unlocking the Full Landscape of Earth Observation with Agents
- SwiftDossier: Tailored Automatic Dossier for Drug Discovery with LLMs and Agents
- Text-to-SQL in the Wild: A Naturally-Occurring Dataset Based on Stack Exchange Data
- PathVQA: 30000+ Questions for Medical Visual Question Answering
- Measuring Massive Multitask Language Understanding
- LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code
- SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
- (QA)$^2$: Question Answering with Questionable Assumptions
- LAB-Bench: Measuring Capabilities of Language Models for Biology Research
- Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflows
- Show Me What and Where has Changed? Question Answering and Grounding for Remote Sensing Change Detection
- MMSci: A Dataset for Graduate-Level Multi-Discipline Multimodal Scientific Understanding
- SceMQA: A Scientific College Entrance Level Multimodal Question Answering Benchmark
- Know What You Don't Know: Unanswerable Questions for SQuAD
Related papers
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Agents' Last Exam
- The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
- MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
- Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems