Benchmark Designers Should "Train on the Test Set" to Expose Exploitable Non-Visual Shortcuts

Name: Benchmark Designers Should "Train on the Test Set" to Expose Exploitable Non-Visual Shortcuts
Uploaded: 2026-10-11T07:04:22.489321+00:00
Duration: 155 s
Description: Benchmark designers should proactively test their creations by 'training on the test set' to find non-visual shortcuts. The study used two diagnostic methods, TsT-LLM and TsT-RF, to quantify how much a benchmark can be solved using only non-visual information. Results showed widespread vulnerability in benchmarks like CV-Bench and VSI-Bench.

Benchmark designers should proactively test their creations by 'training on the test set' to find non-visual shortcuts. The study used two diagnostic methods, TsT-LLM and TsT-RF, to quantify how much a benchmark can be solved using only non-visual information. Results showed widespread vulnerability in benchmarks like CV-Bench and VSI-Bench.

Open the full episode