FinReportBench: Measuring and Improving Institution-Grade Financial Report Generation

arXiv:2608.04374 · cs.CL, cs.AI · Submitted 2026-08-05 · Read on arXiv

Yinghao Tang, Tan Zhenwei, Yiyao Wang, Wanli Gu, Xiaolu Zhang, Jun Zhou, Wei Chen

cs.CL, cs.AI

Submitted: 2026-08-05

Comments: 9 pages, 9 figures

Code: https://github.com/MisterBrookT/finreportbench

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Terminology

• Benchmark — 40× in this paper · explained in 2 episodes
• Trace — 6× in this paper · explained in 2 episodes
• External Validation — 5× in this paper · explained in 2 episodes
• Benchmarking — 3× in this paper · explained in 4 episodes
• Distillation — 3× in this paper · explained in 2 episodes
• Grounding — 3× in this paper · explained in 3 episodes
• Calibration — 2× in this paper · explained in 9 episodes
• Composition — 2× in this paper · explained in 2 episodes
• Generator — 2× in this paper · explained in 2 episodes
• LLM-as-a-Judge — 2× in this paper · explained in 4 episodes
• Accuracy — 1× in this paper · explained in 1 episode
• Agentic Framework — 1× in this paper · explained in 7 episodes
• Cosine Similarity — 1× in this paper · explained in 3 episodes
• Cross-Model Transfer — 1× in this paper · explained in 2 episodes
• Evaluation Framework — 1× in this paper · explained in 3 episodes
• Hierarchical Framework — 1× in this paper · explained in 2 episodes
• Multi-Agent System — 1× in this paper · explained in 7 episodes
• Multimodal Reasoning — 1× in this paper · explained in 5 episodes
• Papers — 1× in this paper · explained in 3 episodes
• Provenance — 1× in this paper · explained in 5 episodes
• Reinforcement Learning — 1× in this paper · explained in 4 episodes
• Verification — 1× in this paper · explained in 2 episodes

Sources

• EvoSkill: Automated Skill Discovery for Multi-Agent Systems
• OpenScholar: Synthesizing Scientific Literature with Retrieval-augmented LMs
• ResearchAgent: Iterative Research Idea Generation over Scientific Literature with Large Language Models
• You Don't Know Until You Click:Automated GUI Testing for Production-Ready Software Evaluation
• Mind2Web 2: Evaluating Agentic Search with Agent-as-a-Judge
• From Raw Experience to Skill Consumption: A Systematic Study of Model-Generated Agent Skills
• FinanceBench: A New Benchmark for Financial Question Answering
• Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning
• The BiGGen Bench: A Principled Benchmark for Fine-grained Evaluation of Language Models with Language Models
• SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Tasks
• FinMME: Benchmark Dataset for Financial Multi-Modal Reasoning Evaluation
• UniChart: A Universal Vision-language Pretrained Model for Chart Comprehension and Reasoning
• WebGPT: Browser-assisted question-answering with human feedback
• VIS-Shepherd: Constructing Critic for LLM-based Data Visualization Generation
• SKILLFOUNDRY: Building Self-Evolving Agent Skill Libraries from Heterogeneous Scientific Resources
• JudgeBench: A Benchmark for Evaluating LLM-based Judges
• ViviDoc: Generating Interactive Documents through Human-Agent Collaboration
• Demonstrating chart-plot: Closing the Last Mile of Academic Chart Generation
• BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agents
• WebDancer: Towards Autonomous Information Seeking Agency

Related papers

• Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
• Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
• Subliminal Steering: Stronger Encoding of Hidden Signals
• MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
• The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
• Untangling the Mechanisms of Misleading Context in Medical Question Answering