ReuScope: Auditing Data Reuse and Evaluation Sensitivity in Capsule Vision 2024
cs.AI, cs.CV
Submitted: 2026-10-30
Updated: 2026-10-30
Comments: Accepted at NeurIPS 2026, Track on Evaluations and Datasets. 39 pages, 3 figures. Code and data: https://github.com/eunseobchoi/ReuScope
Code: https://github.com/JohannesBuchner/imagehash
License: http://creativecommons.org/licenses/by/4.0/
The gist: We develop ReuScope, a protocol for auditing data reuse and evaluation sensitivity, using Capsule Vision 2024 as a case study.
Terminology
Abstract
We develop ReuScope, a protocol for auditing data reuse and evaluation sensitivity, using Capsule Vision 2024 as a case study. It combines measurements of data relationships with two comparisons: changing the evaluation population while holding predictions fixed, and changing training data while holding evaluation fixed. Byte-identical files cross the public training-validation boundary, and all 41 Kvasir-Capsule recordings that contribute frames occur in both partitions. Shared source recordings establish cross-split overlap; missing cross-recording patient identifiers prevent counting unique shared patients. Across ten paired seeds, removing all training rows attributed to Kvasir-Capsule, plus five rows from other sources flagged as near-matches to it, lowers public-validation balanced accuracy from 82.5% to 61.2%. Evaluating the same fitted models on the validation rows of the other sources reverses the comparison: removal improves balanced accuracy by 1.38 percentage points. The public-validation drop measures sensitivity to source removal, not an amount of leakage-induced inflation. Matched controls show that sample size and class counts alone do not explain the observed score differences. An exploratory experiment removes the training neighbors of validation frames that have close training matches, or equally many distant training rows, and yields a difference-in-differences estimate, relative to matched unflagged frames, of -0.355 percentage points in true-class probability; the paired-t interval for accuracy spans zero. The resulting procedure distinguishes observed reuse from conditional performance sensitivity and specifies the population, class support and uncertainty unit for each conclusion.
Sources
- Capsule Vision 2024 Challenge: Multi-Class Abnormality Classification for Video Capsule Endoscopy
- Domain-Adaptive Pre-training of Self-Supervised Foundation Models for Medical Image Classification in Gastrointestinal Endoscopy
Related papers
- Multi-Objective Structured Pruning of LLMs for Latency and Model Size Optimization
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Embeddings for Preferences, Not Semantics
- Can We Trust Item Response Theory for AI Evaluation?
- Diagnosing Task Insensitivity in Language Agents