Agentick: A Unified Benchmark for General Sequential Decision-Making Agents
cs.AI
Submitted: 2026-05-07
Updated: 2026-09-25
Comments: Published at NeurIPS 2026 Evaluations & Datasets Track
Project page: https://roger-creus.github.io/agentick/board
License: http://creativecommons.org/licenses/by/4.0/
The gist: AI agent research spans a wide spectrum: from RL agents that learn from scratch to foundation model agents that leverage pre-trained knowledge, yet no unified benchmark enables fair comparison across
Terminology
Abstract
AI agent research spans a wide spectrum: from RL agents that learn from scratch to foundation model agents that leverage pre-trained knowledge, yet no unified benchmark enables fair comparison across these approaches. We present Agentick, a benchmark for sequential decision-making agents designed to evaluate RL, LLM, VLM, hybrid, and human agents on common ground and to power research on the fundamental challenges of sequential decision-making. Agentick provides 37 procedurally generated tasks across six capability categories, four difficulty levels, and five observation modalities, all exposed through a single Gymnasium-compatible interface. The benchmark ships with a Coding API, oracle reference policies for all tasks, pre-built SFT datasets, a composable agent harness, and a live leaderboard. An evaluation spanning 27 configurations and over 90,000 episodes reveals that no single approach dominates: GPT-5 mini leads overall at 0.309 oracle-normalized score while PPO dominates planning and multi-agent tasks; the reasoning harness multiplies LLM performance by 3-10x; and ASCII observations consistently outperform natural language. These findings highlight the substantial room for improvement that remains across all agent paradigms. Agentick's capability-decomposed, multi-modal design provides the empirical infrastructure needed to drive progress toward general autonomous agents, both as an evaluation framework and as a training ground for RL post-training of foundation models in truly sequential environments.
Sources
- Do As I Can, Not As I Say: Grounding Language in Robotic Affordances
- ARC-AGI-3: A New Challenge for Frontier Agentic Intelligence
- ARM-FM: Automated Reward Machines via Foundation Models for Compositional Reinforcement Learning
- Stable Gradients for Stable Learning at Scale in Deep Reinforcement Learning
- Evaluating Large Language Models Trained on Code
- Training Verifiers to Solve Math Word Problems
- DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
- Measuring Mathematical Problem Solving With the MATH Dataset
- BALROG: Benchmarking Agentic LLM and VLM Reasoning On Games
- Proximal Policy Optimization Algorithms
- REASONING GYM: Reasoning Environments for Reinforcement Learning with Verifiable Rewards
- DeepMind Control Suite
- Gymnasium: A Standard Interface for Reinforcement Learning Environments
Related papers
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Agents' Last Exam
- The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
- MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
- Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems