Where It Breaks: Diagnosing Perception, Memory, and Decision Failures of a Vision-Language Agent in Pac-Man

arXiv:2611.06704 · cs.LG · Submitted 2026-11-04 · Read on arXiv

cs.LG

Submitted: 2026-11-04

Updated: 2026-11-16

Comments: 27 pages, 7 figures

License: http://creativecommons.org/licenses/by/4.0/

The gist: Spatial agents must interpret observations, maintain consistent spatial memory and make decisions over long interactions.

Terminology

Abstract

Spatial agents must interpret observations, maintain consistent spatial memory and make decisions over long interactions. We use Pac-Man as a testbed to study the training challenges behind these capabilities in a Qwen3.5-9B-based agent. Our research trajectory progresses from primitive-action RL on text-based mazes to visual inputs with an internal-state-assisted harness, then toward observation-driven memory and joint policy--harness improvement. The main analysis focuses on text-based maze RL; supplementary sections report visual-adaptation and harness experiments and discuss subsequent directions. Text-based maze RL requires curriculum and training-recipe design to overcome cold starts; normal-start clear rates on eight held-out mazes rise from 0% to a peak of 81.25%. In the second step, model adaptation and harness improvements raise the clear rate from 26.6% to 96.9% on one fixed map with three extra lives. These results highlight three development lessons: use curricula to address cold starts, establish visual grounding before language adaptation, and treat harness design as a determinant of agent performance. They provide practical guidance for subsequent VLM training and spatial-agent development.

Sources

Related papers