Perception Test 2026: Challenge Summary and Extension to City-scale Audio-Visual Reasoning

The gist The winning solutions in the newly added city-scale tracks demonstrated that complex spatial and multimodal reasoning can be solved by expensive agentic pipelines, but remains difficult for multimodal models used standalone

Challenge Overview

The fourth edition of the Perception Test challenge was organized as a workshop at the European Conference on Computer Vision (ECCV) 2026 in Malmö, Sweden This edition focused on spatial intelligence and featured four different tracks: unified multiple-choice videoQA and grounded videoQA from the original Perception Test benchmark, alongside two new tracks based on city-scale walking-tour videos (KilometerAudio and KilometerVision) The challenge setup involved validation and test phases for each track, with test servers becoming available starting in midAugust 2026 for KilometerAudio and KilometerVision Participants were required to report the evaluation mode (zero-shot, few-shot, or fine-tuned) and submit a short technical report

KilometerVision Benchmark

KilometerVision is a recent benchmark that probes city-scale spatial capabilities in frontier models using real-world walking tour Youtube videos, up to 10 minutes long, corresponding to about 1KM walking distance The tasks use multiple-choice videoQA format, with three different task configurations: standard multiple-choice video QA with five text options; multiple-choice video QA where the five options are images (e.g. candidate route maps, see example in Figure 1); and multiple-choice video QA where the question includes an image (e.g. a landmark whose distance to the final camera position must be estimated, see example in Figure 2) The benchmark includes seven types of questions: landmark recognition, distance to landmark, compass, loop closure, route summary, map trace (with or without text labels on the map), and Euclidean distance

KilometerAudio Benchmark

To assess models’ audio-visual capabilities in large-scale environments, the KilometerAudio benchmark is a novel benchmark introduced here for the first time It aims to test the ability of foundational models, including agentic approaches, to reason about the interlink between audio and visual modalities in very long videos – i.e. videos captured over walking for several kilometers around an urban environment Natural sounds that are automatically captured in these public YouTube videos include sounds of human-made objects (cars, motorbikes, church bells, etc.), natural sounds (birds chirping and dogs barking), speech as well as acoustic urban pollution Reasoning about an in-view visual along with an out-of-view audio as well as the length of the videos make this benchmark particularly unique The test set contains 500 five-option questions over 218 publicly available hour-long Youtube walking-tour videos (1–3 h) All the questions in KilometerAudio were collected with human raters and checked by our research team The raters were instructed to identify recognisable sounds in the videos and formulate questions that require both the audio and visual modalities to be answered: identifying sound sources in space, localising or timing audio events, counting them, recognising songs or speech; see Figure 3

Results Summary

The performance on the remaining three tracks almost saturated (> 80% accuracy), which is remarkable especially given that two of the tracks, the city-scale tracks, have been run this year for the first time Almost all top solutions were zero-shot agentic pipelines built around proprietary or open-source VLMs (Gemini, GPT, Doubao-Seed, Qwen), often combined with ensembling or self-consistency metrics Among the awarded entries, only the unified multiple-choice winner fine-tuned a model on the Perception Test training data The best performing methods in the new city-scale tracks relied on expensive and slow agentic tool use, audio indexing, or explicit geometric reconstruction, but obtained almost perfect performance In conclusion, large-scale audio-visual and spatial reasoning can be solved by systems that orchestrate models with explicit tools, but remains an open problem for multimodal models used standalone

Track Specific Results

**- Unified multiple-choice video QA The winning team, NjustKMG, used a question-type-aware weighted ensemble of three models: Qwen3-VL-32B finetuned on the Perception Test training set (with a curriculum stage on the hardest question types and option shuffling to reduce position bias), and the Doubao-Seed-2.0-Pro and Gemini 3.1 Pro APIs queried zero-shot with question-type specific prompts; the ensemble weights are derived from per-type validation accuracy The runner-up, F423, used a single zero-shot model, Doubao-Seed-2.0 Lite, with evidence first prompting (extracting the visual, temporal, and audio evidence before comparing the options), adaptive frame sampling (1–2 fps depending on the question type), and strict answer validation with at most one re-query The winning team improved significantly on the questions added in 2025 (86.2% vs 65.4% for the best 2025 model) compared to improvement on the original set (92.2% vs 83.7%) **

**- Grounded video QA The winning team, Banana Bats, proposed Gemini-SAM3: Gemini 3.7 Flash, used zero-shot, builds a tracking plan (target identities, visibil frequency baseline The runner-up Huawei SpaceMind used GPT5.6 Sol zero-shot to identify the answer objects and produce complete box tracks, then refined only the box coordinates, never the object identities, using a second independent track **

**- KilometerVision The winning team, ohmyyuan, proposed evidence-log video agents: Gemini 3.7 Flash builds a timestamped log of street names, heading changes, and landmarks from frame montages and keyframes, and answers with a family-specific playbook The best performing entry not eligible for a prize obtained the best overall score (0.905) with an agentic pipeline that explored the validation set and a small fraction (about 10%) of the unlabelled test questions to learn a strategy for solving the task on the full test set **

**- KilometerAudio The winning team, IUCV, proposed AVRA, a training-free audio-visual reasoning agent built on Gemini 3.7 Flash The runner-up WW11 proposed a zero-shot tool using an agent in which Gemini 3.1 Pro plans the search and reasons over at most 48 frames or crops per question, using offline indexes of timestamped speech transcripts (Whisper) and CLAP based sound event retrieval The best performing entry obtained again the best overall score (0.864) with a similar agentic pipeline as in the KilometerVision track, but using signal-processing tools instead of 3D reconstruction, to compile a strategy for solving the tasks **

**- Best Generalist award The Qwen3.8-Max model makes every final answer and box decision, without task specific detectors, trackers, or geometry solvers Although the model trailed behind the per-track winners, this is the first submission across the four editions that shows a unified model with satisfactory performance across all the tracks The results of the generalist model were: 0.896 top-1 in multiplechoice video QA (5th place, 98% relative to the top solution), 0.465 HOTA in grounded video QA (13th place, 69% of top), 0.604 in KilometerVision (9th place, 80% of top), and 0.

1. Grounding large-scale spatial reasoning via agentic pipelines: The winning solutions demonstrated that complex spatial and multimodal reasoning can be solved by expensive agentic pipelines, but remains difficult for multimodal models used standalone. This suggests improving AI systems by designing expensive agentic pipelines that orchestrate specialized models, allowing them to solve city-scale tasks like those in KilometerVision.

2. Audio-visual interlink reasoning in long videos: The KilometerAudio benchmark tests the ability of foundational models, including agentic approaches, to reason about the interlink between audio and visual modalities in very long videos. Systems can be improved by incorporating methods like those used by IUCV (AVRA, a training-free audio-visual reasoning agent built on Gemini 3.7 Flash) which uses an explicit evidence log and derives an answer before reading the options.

3. Robust spatial tracking and geometric estimation: For KilometerVision, systems can be improved by combining VLM perception with explicit geometric solvers, as demonstrated by the winning team's approach using Gemini 3.7 Flash builds a timestamped log of street names, heading changes, and landmarks alongside deterministic estimators (optical-flow yaw odometry and visual place recognition).

4. Generalist model development for cross-track performance: The Best Generalist entry showed promise by using a single training-free approach to achieve satisfactory performance across all the tracks. This indicates that future systems should focus on developing unified models that maintain strong baseline accuracy (e.g., 0.896 top-1 in multiple-choice video QA) while showing improvement in city-scale tasks.

5. Question-type aware ensemble methods: The Unified multiple-choice video QA winner used a question-type aware weighted ensemble of three models, suggesting that AI systems should be improved by dynamically selecting and weighting specialized models based on the specific reasoning required by the task, as evidenced by their 91.4% accuracy.

Continuing the Perception Test challenge series, we organised the fourth edition as a workshop at the European Conference on Computer Vision (ECCV) 2026 in Malmö, Sweden. This edition focused on spatial intelligence and featured four different tracks: unified multiple-choice videoQA and grounded videoQA from the original Perception Test benchmark, alongside two new tracks based on city-scale walking-tour videos (KilometerAudio and KilometerVision). In this report, we describe the new benchmarks used for the city-scale tracks and summarise the winning solutions across all tracks, including a generalist model that competed across all tracks with satisfactory performance. The winning solutions in the newly added city-scale tracks demonstrated that complex spatial and multimodal reasoning can be solved by expensive agentic pipelines, but remains difficult for multimodal models used standalone.