GemNav: Discrete-Token Visual Robot Navigation using a Multimodal Large Language Model
GemNav adapts a frozen Multimodal Large Language Model (MLLM) for waypoint navigation using Low-Rank Adaptation (LoRA) on its language tower only, avoiding new visual encoders or regression heads. It uses discrete tokens to unify waypoints and stop signals, achieving zero-shot success in four unseen environments with minimal training data.