VLMs for Videogame Data Annotation

The paper, authored by Katrin Schmid (NVIDIA, Australia) and Iuri Frosio (NVIDIA, Italy), investigates the use of Vision Language Models (VLMs) and AI agents for annotating video game frame sequences with dense, human-defined reward signals. The authors note that while VLMs and AI agents demonstrate a remarkable understanding of the real world, their adoption in video games is on the other hand limited by the extreme variability of the synthetic scenarios and their poor compliance with real-world physics. The task has several potential applications including conditioned training and offline reinforcement learning.

The motivation stems from a significant domain gap: "AI models have been mostly trained on real-world data and therefore suffer from a significant domain gap: for instance, video game physics may allow jumping for meters or crashing a car without reporting any damage, which is in contrast with the common-sense inductive bias instilled in AI models. The authors demonstrate with examples (Fig. 1) that VLMs like ChatGPT-5.4, Qwen-3.5, and Nemotron2-12B make various errors on Wreckfest annotations: they may misinterpret the scene visual features"; need a more precise prompt; hallucinate other vehicles and wrongly assume the car to be in optimal position; wrongly associate the displayed damage report with a recent impact against the barrier; or correctly identify the damage as pre-existing yet still failing to detect the barrier contact from visual evidence.

The paper's contributions are: (1) comparing several VLMs on Visual Question Answering for Trackmania and Wreckfest game sequences; (2) introducing a linear mix model to improve annotation accuracy (the Reward Annotation Model, or RAM); and (3) analyzing the effects of prompt optimization, input sequence length and resolution, and question batching on accuracy and cost.

Method. The goal is to automatically annotate long video game sequences with N expert-defined, dense rewards Rj, each associated with a specific question Qj. The authors use a premise prompt P to provide context and ask the VLM to output a probability in the 0–100% range. For Trackmania, the three questions are: Q1 is the red-and-white car in the middle of the road?, Q2 is the red-and-white car moving forward?, and Q3 is the red-and-white car hitting the lateral barriers? They found that annotating sequences of five frames at 20Hz to work well for Trackmania, and for long clips they annotate 5-frame sequences with 1 frame overlap using the VLM and interpolate the output probabilities to get a continuous reward signal.

The RAM model exploits correlations between rewards: Rewards often show some form of correlation: for instance, a car is unlikely to be in the middle of the road (Q1) when it hits a lateral barrier (Q3). RAM learns a linear combination of VLM outputs on a small human-annotated reference set: aRAM j(t) = Σ k w k,j aVLM k(t) + b j. The weights are learned by either minimization of an L2 loss or by maximization of a soft Fβ score. The authors report results for F1, as we found that the L2 loss (or Fβ for β≠1) generally achieve worse results. Unlike Snorkel [8], which aggregates weak labels by voting, RAM learns a simple linear combination of VLM outputs on a small human-annotated reference set.

For prompt optimization, the authors used the DSPy framework with MIPROv2, which performs Bayesian optimization over the prompt space; each question Qj ends up with a premise Pj that differs from the original one and the optimization is model-specific. For question batching, instead of one call per question, the VLM is asked to output a vector that answers all the questions Qj j=1...N at the same time to minimize the number of calls.

Results. The testing dataset was a small dataset of T = 5500 frames (approx. 4m 36s), later split into 50% training and 50% validation, of Trackmania, a racing video game with clean synthetic visuals. The dataset was human-annotated in under an hour.

Plain annotation. Table II shows that ChatGPT-5.4 leads on average over all other VLMs, but its Terms of Service forbid downstream AI training. From a practical point of view, thus, Qwen3.5 and Kimi-k2.5 represent better solutions. Regarding resolution, more pixels lead to better results, but also higher token usage. The authors emphasize that a low correlation coefficient with high precision (or recall) identifies degenerate cases where the VLM has a constant output and the dataset is unbalanced.

RAM evaluation. Table III shows significant improvements provided by RAM over plain annotation for any VLM considered. Furthermore, RAM reduces the gap between different VLMs. The comparison with Snorkel shows: "Snorkel-LM ties RAM on Q1 (F1 = 90.2, 89.9 for RAM) but collapses on the sparse Q3 (F1 = 0.0, 76.2 for RAM), confirming that RAM's supervised aggregation strategy is superior when classes are imbalanced; Snorkel-MV underperforms across all questions."

Question batching. "Batching reduces the tokens, but interference between the questions in the same prompt affects the quality of annotation, generally reducing F1. Interestingly enough, applying RAM to the batched questions pushes F1 up again, achieving the best compromise between quality and token usage."

Sequence length. "Fig.4 shows F1 as a function of the length of the input sequence: longer input sequences require more tokens per annotation with F1 decreasing, likely because the correct answer to the given question may vary per frame along the sequence itself, thus decreasing the confidence of the VLM."

Prompt optimization. "Our experiments with MIPROv2 delivered contrasting results: Table IV shows that F1 improves for Q1 and Q3, but the original prompt is better for Q1 after RAM. The effect of MIPROv2 on Q2 is negligible, but it leads to some improvement after RAM. Overall we found the output of MIPROv2 to be poorly predictable especially in cases of unbalanced datasets, as for Q2 and Q3."

Wreckfest. Wreckfest is a demolition-derby game with complex multi-car scenes and realistic damage rendering. Table V "confirms that RAM consistently improves the annotation quality, but comparison against Tables II and III highlights that VLMs may struggle on video games with richer visual features (when compared to Trackmania) or less favorable camera configuration (first-person view as for Wreckfest in Fig. 1 vs. third-person view as for Trackmania in Fig. 2)."

Conclusion. The authors conclude: "We studied reward annotation for video games through VLMs and shown that the VLM choice, question batching, frame resolution and sequence length impact the annotation quality and the token consumption. By leveraging the correlation between the VLM answers (via a simple linear model like RAM) we find a better compromise between the annotation cost and its quality. Other approaches, like automatic prompt optimization, provide less consistent improvements."

As limitations, they acknowledge: the significance of our tests is so far limited by the small single-annotator reference set: this is why we plan to study larger sets and more games. Other factors include "commercial constraints of some VLMs whose terms of service prevent downstream AI training; furthermore, content filters can block violent scenes — broadly defined from car accidents to war scenes — that are common in gaming. The authors conclude that the fact that VLMs still struggle on complex video game scenarios, as shown for Wreckfest, suggests that significant room for improvement still exists and more research is needed to use VLMs as reliable video game annotators."

Improved AI systems can incorporate the following specific enhancements based on this paper:

1. Learnable linear correction layer (RAM): Add a lightweight linear model that maps raw VLM probability outputs to calibrated reward scores, trained on a small human-annotated reference set (e.g., 50% of frames). This improves F1 significantly for all VLMs, reduces variance across models, and prevents degenerate constant-output collapse, especially for sparse/imbalanced questions. The system can then output dense reward signals with confidence estimates.

2. Adaptive sequence length selection: Use short input sequences (e.g., 5 frames at 20Hz) with 1-frame overlap and interpolation to produce dense rewards, rather than feeding longer clips. This improves annotation accuracy because long sequences confuse VLMs when the answer varies per frame. The improved system can automatically chunk long gameplay into overlapping short windows and interpolate probabilities to reconstruct a continuous reward.

3. Batched question handling with post-hoc correction: When batching multiple reward questions into one VLM call to save tokens, apply the learned linear correction (RAM) to the batched outputs. This recovers most of the F1 lost due to inter-question interference, achieving the best quality/cost trade-off.

4. Resolution and cost trade-off control: The system can dynamically choose frame resolution based on the difficulty of the visual task (e.g., lower resolution for simple Trackmania-style scenes, higher for Wreckfest-like cluttered scenes) by monitoring validation-set F1 and token usage, optimizing cost without sacrificing annotation quality.

5. Selective prompt optimization: Instead of blindly applying automatic prompt optimization (MIPROv2), the system should evaluate candidate prompts on a small validation set for each reward question and for the post-RAM output. It can then choose the prompt that maximizes final calibrated F1, avoiding cases where raw F1 improves but RAM-corrected F1 degrades.

6. Game-aware domain adaptation: For video games with unrealistic physics (e.g., no damage on crashes, extreme jumps), the system can include a game-specific physics prior in the prompt, explicitly stating deviations from real-world physics. Alternatively, fine-tune the VLM on a small set of synthetic gameplay frames to reduce the real-to-synthetic domain gap.

7. Open-weight model preference for trainable pipelines: Use VLMs whose terms of service allow downstream AI training (e.g., Qwen, Kimi) instead of commercial ones (e.g., ChatGPT) when the annotation output is intended to train another AI model. The system can automatically enforce this license-aware selection.

8. Robust handling of sparse positive classes: For events that rarely occur (e.g., hitting lateral barriers), the system should avoid unsupervised majority voting (e.g., Snorkel) and instead use supervised linear aggregation (RAM), which prevents total collapse (F1=0) and maintains meaningful recall/precision balance.

9. Visual richness detection and mode switching: The system can analyze game scenes (e.g., object density, camera perspective, damage rendering) and select different annotation strategies: for third-person, clean-synthetic scenes use standard VLM+RAM; for first-person, multi-car, realistic-damage scenes, use higher resolution, longer prompts with explicit object-count directives, or combine VLM outputs with a lightweight object detector to reduce hallucination of other vehicles.

10. Interpolated dense reward generation for offline RL: Given sparse VLM annotations on overlapping windows, the system can output a continuous, temporally smooth reward signal (via linear interpolation) suitable for conditioned training and offline reinforcement learning, with per-timestep uncertainty estimates derived from RAM's weight vector and the VLM's output variance.

These improvements enable an AI system to automatically annotate hours of gameplay with human-aligned, dense reward signals at lower cost, higher reliability, and without violating model license restrictions — making it practical for training game-playing agents.