Frontier Autolab: Organizational Memory, Adversarial Dissent and Temporal Leakage in Multi-Agent LLM Firms Across Fifty Years of Technological Change
cs.CE, cs.AI
Submitted: 2026-09-29
Updated: 2026-09-29
Comments: 17 pages, 6 figures, 9 tables. Code and data: https://github.com/LoopGlitch26/Frontier-Autolab
Code: https://github.com/LoopGlitch26/Frontier-Autolab
License: http://creativecommons.org/licenses/by/4.0/
The gist: Multi-agent LLM systems are increasingly structured like organizations, with roles, critics and shared memory, yet they are evaluated on tasks that last minutes.
Abstract
Multi-agent LLM systems are increasingly structured like organizations, with roles, critics and shared memory, yet they are evaluated on tasks that last minutes. We ask how such an organization behaves when the ground it stands on keeps moving. Frontier Autolab is a long-horizon testbed in which one simulated firm, voiced by sixteen role personas and a dedicated Red Team, must re-found itself in nine technology eras from 1990 to 2040. Each era is temporally gated: the firm decides from a dated briefing, a historian-judge then reveals what happened and scores the decision on a five-dimension rubric, and lessons enter a persistent Playbook. Six eras are scored against history, one against the live market and two are open forecasts. Across four trajectories (36 era decisions, 180 subscores) we find a consistent foresight-commitment gap: in all 24 historically scored eras the judge rated the firm's recognition of the coming shift above its choice of where to build (mean gap 1.9 points on a 10-point scale), because boards chose the layer their existing assets could reach. Organizational design shaped long-run character. A Red Team armed with numeric kill gates produced fifty years of gated pilots and no product, and the rubric rated this firm highest; firms whose memory stored market-structure lessons pivoted every era, while a firm whose memory stored only validation procedure kept one method throughout. We also show why such results are hard to trust. Scores rise across eras in every run while the judge's own hindsight subscore falls (within-run r = -0.58), so apparent learning is confounded with recall of history, and we trace further distortions to self-judging, briefing selection and score aggregation. We release all records and an API harness, and specify fictional and post-cutoff eras that would turn the testbed into a benchmark.
Sources
- AgentVerse: Facilitating Multi-Agent Collaboration and Exploring Emergent Behaviors
- Dated Data: Tracing Knowledge Cutoffs in Large Language Models
- Improving Factuality and Reasoning in Language Models through Multiagent Debate
- Assessing Look-Ahead Bias in Stock Return Predictions Generated By GPT Sentiment Analysis
- Time Travel in LLMs: Tracing Data Contamination in Large Language Models
- Large Language Model based Multi-Agents: A Survey of Progress and Challenges
- Approaching Human-Level Forecasting with Language Models
- MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework
- Large Language Models as Simulated Economic Agents: What Can We Learn from Homo Silicus?
- ForecastBench: A Dynamic Benchmark of AI Forecasting Capabilities
- Measuring AI Ability to Complete Long Software Tasks
- CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Society
- Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate
- Pitfalls in Evaluating Language Model Forecasters
- LLM Evaluators Recognize and Favor Their Own Generations
- Generative Agents: Interactive Simulacra of Human Behavior
- ChatDev: Communicative Agents for Software Development
- NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmark
- Reflexion: Language Agents with Verbal Reinforcement Learning
- Voyager: An Open-Ended Embodied Agent with Large Language Models
Related papers
- Constrained Sensing and Reliable State Estimation with Shallow Recurrent Decoders on a TRIGA Mark II Reactor
- A finite-element-inspired bipartite graph learned simulator for manufacturability assessment in large-deformation sheet forming
- Non-linear mechanical field reconstruction coupling recurrent neural networks with physics-informed graph neural networks
- Neural-Parameterized Cellular Automata for Wildfire Spread
- A mesh-based neural energy method for the simulation of heterogeneous composites
- Evidence-Unit Fairness and the Limits of Query-Adaptive Sparse-Dense Fusion in Financial Document Retrieval