EComAgentBench: Benchmarking Shopping Agents on Long-Horizon Tasks with Distributed Hidden Intent

arXiv:2606.17698 · cs.AI, cs.CL · Submitted 2026-06-16 · Read on arXiv

cs.AI, cs.CL

Submitted: 2026-06-16

Updated: 2026-09-02

Comments: 16 pages, 4 figures; accepted to the EMNLP 2026 Industry Track (camera-ready version)

Code: https://github.com/Morizeyao/EComAgentBench_

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Terminology

• Benchmark — 40× in this paper · explained in 2 episodes
• Accuracy — 17× in this paper · explained in 1 episode
• Benchmarking — 5× in this paper · explained in 3 episodes
• Verification — 4× in this paper · explained in 2 episodes
• Core — 3× in this paper · explained in 2 episodes
• Failure Modes — 2× in this paper · explained in 3 episodes
• Personalization — 2× in this paper · explained in 2 episodes
• LLM judge — 1× in this paper · explained in 2 episodes
• LLM-as-a-Judge — 1× in this paper · explained in 3 episodes
• Structural Constraints — 1× in this paper · explained in 2 episodes

Sources

• Bridging Language and Items for Retrieval and Recommendation: Benchmarking LLMs as Semantic Encoders
• VitaBench: Benchmarking LLM Agents with Versatile Interactive Tasks in Real-world Applications
• ShopSimulator: Evaluating and Exploring RL-Driven LLM Agent for Shopping Assistants
• AssistantBench: Can Web Agents Solve Realistic and Time-Consuming Tasks?
• DeepPlanning: Benchmarking Long-Horizon Agentic Planning with Verifiable Constraints
• WebArena: A Realistic Web Environment for Building Autonomous Agents

Related papers

• MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
• Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
• Agents' Last Exam
• The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
• MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
• Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems