Goal Retrieval versus Action-Value Ordering: Contrastive Critics under Best-of- K Selection
The paper investigates using contrastive critics for best-of-K action selection in goal-conditioned reinforcement learning. While these critics are good at ranking actions based on goal compatibility, they lack reliable value calibration, meaning their rankings don't accurately reflect the actual expected reward. The study shows that while raw and cosine critics achieve high retrieval accuracy, the lack of value ordering leads to poor performance during action selection.