CorporateBench: Large-Scale Q&A Benchmarking with Temporal Knowledge Bases
cs.AI, cs.CL, cs.IR, cs.LG
Submitted: 2026-08-27
Updated: 2026-08-27
Comments: Accepted to EMNLP Findings
License: http://creativecommons.org/licenses/by-nc-sa/4.0/
The gist: LLMs are increasingly able to answer complex questions about enterprise-scale document collections.
Terminology
Abstract
LLMs are increasingly able to answer complex questions about enterprise-scale document collections. But evaluation is hard: companies don't want to share internal communications, and synthetic datasets have been overly simple. We present CorporateBench (CB), a human-validated multi-task Q&A benchmark whose scale approaches the conditions LLMs encounter in corporate communication networks, with evaluation corpora surpassing 230,000 documents. CB evaluates LLMs across two dimensions (information extraction and knowledge base querying) through four synthetically generated firms ranging from 12 to 10,000 employees. Each corpus is sampled from a temporally evolving knowledge base describing a consistent world, guaranteeing cross-document logical consistency even across hundreds of thousands of documents. We evaluate five LLMs on CB, revealing increasingly poor performance as input size approaches realistic scales. CB provides LLM developers a metric for corporate communication reasoning, filling a crucial gap in the benchmarking ecosystem.
Sources
- L-Eval: Instituting Standardized Evaluation for Long Context Language Models
- Longformer: The Long-Document Transformer
- Current and Future Use of Large Language Models for Knowledge Work
- OneEval: Benchmarking LLM Knowledge-intensive Reasoning over Diverse Knowledge Bases
- Too Long, Didn't Model: Decomposing LLM Long-Context Understanding With Novels
- Benchmarking Deep Search over Heterogeneous Enterprise Data
- RULER: What's the Real Context Size of Your Long-Context Language Models?
- WorkArena: How Capable Are Web Agents at Solving Common Knowledge Work Tasks?
- LawBench: Benchmarking Legal Knowledge of Large Language Models
- PhantomWiki: On-Demand Datasets for Reasoning and Retrieval Evaluation
- The Llama 3 Herd of Models
- Advancing Transformer Architecture in Long-Context Large Language Models: A Comprehensive Survey
- Enhancing Question Answering for Enterprise Knowledge Bases using Large Language Models
- Scaling Laws for Neural Language Models
- One Thousand and One Pairs: A "novel" challenge for long-context language models
- Learning to Learn Semantic Parsers from Natural Language Supervision
- Multi-Relational Question Answering from Narratives: Machine Reading and Reasoning in Simulated Worlds
- Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
- GPT-4o System Card
- EnronQA: Towards Personalized RAG over Private Documents
Related papers
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Agents' Last Exam
- The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
- MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
- Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems