Causal Evidence Extraction and Triangulation in Crisis Reports using Large Language Models: A ReliefWeb-based Study
Yuanjun Zhang, Mourad Oussalah
cs.CL
Submitted: 2026-08-05
Journal ref: Findings of the Association for Computational Linguistics: ACL 2026, pages 32478-32491, 2026
DOI: 10.18653/v1/2026.findings-acl.1626
License: http://creativecommons.org/licenses/by/4.0/
The gist: Humanitarian reports are long, noisy, and multi-topic, making it difficult to consolidate decision-relevant causal evidence.
Terminology
Abstract
Humanitarian reports are long, noisy, and multi-topic, making it difficult to consolidate decision-relevant causal evidence. We present a ReliefWeb study (2000-2024) and a two-stage Large Language Model (LLM) pipeline that extracts structured intervention-outcome records with direction and strength attributes. Query-conditioned extraction restricts output to a specified intervention class, reducing retrieval-induced over-extraction, while snippet grounding links each relation to supporting text for auditability and classification. In an expert-annotated dataset of 100 reports, the best closed-source LLM achieved a weighted F1 score of 90.73% with strong cost-efficiency, while Llama-3.1-8B with supervised fine-tuning reached 94.15% weighted F1 score. We further propose context-preserving triangulation that aggregates strength-weighted evidence within disaster times source cells, applies Laplace smoothing and equally weights cells to quantify cross-context convergence via a Level-of-Evidence score. Applied to cash assistance, food-related outcomes show strong positive convergence (LoE=0.865) and stable long-horizon trajectories.
Sources
- Qwen Technical Report
- DeepSeek-V3 Technical Report
- EMSDialog: Synthetic Multi-person Emergency Medical Service Dialogue Generation from Electronic Patient Care Reports via Multi-LLM Agents
- The Llama 3 Herd of Models
- GPT-4o System Card
- Faithful-First Reasoning, Planning, and Acting for Multimodal LLMs
- MMFineReason: Closing the Multimodal Reasoning Gap via Open Data-Centric Methods
- Who Stole Your Data? A Method for Detecting Unauthorized RAG Theft
- SynthVLM: Towards High-Quality and Efficient Synthesis of Image-Caption Datasets for Vision-Language Models
- EmoMAS: Emotion-Aware Multi-Agent System for High-Stakes Edge-Deployable Negotiation with Bayesian Orchestration
- EmoDebt: Bayesian-Optimized Emotional Intelligence for Strategic Agent-to-Agent Debt Recovery
- Stable and Explainable Personality Trait Evaluation in Large Language Models with Internal Activations
- STABLE: Efficient Hybrid Nearest Neighbor Search via Magnitude-Uniformity and Cardinality-Robustness
- ExpSeek: Self-Triggered Experience Seeking for Web Agents
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering