Enabling Validation for Robust Few-Shot Recognition

The gist: The core challenge of FSR is data scarcity, extending beyond limited training data to a complete lack of validation data, and this paper introduces Validation-Enabled Stage-wise Tuning (VEST) to resolve this dilemma by harmonizing performance gain and degradation on the few-shot ID data and the retrieved data<ref:2506.04713#pg2>.

Problem Formulation

The problem of robust FSR is formulated by considering a downstream task with a small set of labeled data D, where each class has m labeled examples, i.e., the so-called “m-shot” setting<ref:2506.04713#pg4>. The VLM consists of a visual encoder V (·) and a text encoder T(·), and the image and text embedding features are denoted as xi = V (Ii) and ti = T(yi), respectively<ref:2506.04713#pg4>. Evaluation is performed on ID test sets, which follow the same distribution of the task-specific training data, and OOD test sets, which have distribution shifts but still capture the same classes as the training data<ref:2506.04713#pg4>. The baseline involves finetuning the visual encoder using a cross-entropy (CE) loss LCE over the few-shot training data<ref:2506.04713#pg4>.

Enabling Validation for Checkpoint Selection

The unavailability of a validation set means that finetuned models are especially prone to overfitting, leading to degraded performance on retrieved data which is inherently OOD compared with the task-specific ID training data<ref:2506.04713#pg5>. To address this, the paper introduces a novel validation strategy based on “performance gains” on the ID training data and the retrieved data<ref:2506.04713#pg6>. For each checkpoint at epoch-i, its “performance gain” is computed as ∆tID = accID − mini(accID), and on the retrieved data as ∆tRT = accRT − mini(accRT)<ref:2506.04713#pg6>. The generalization ability of this checkpoint is measured using their harmonic mean, dubbed gF1, defined as gF1t = 2 × ∆tID × ∆tRT / (∆tID + ∆tRT)<ref:2506.04713#pg6>.

Enabling Validation for Layer Selection

To determine which parameters to finetune, the strategy extends to determining k for partially finetuning (PFT) the top-k layers or blocks in a Transformer<ref:2506.04713#pg7>. The validation strategy selects PFT top-1 layers, which provides a good balance between ID and OOD performance and remarkably outperforms linear probing and full finetuning<ref:2506.04713#pg9>. This selection is guided by the gF1 score, which selects the checkpoint that performs robustly well and generalizes to both ID and OOD test data<ref:2506.04713#pg9>.

VEST: Validation-Enabled Stage-wise Tuning

The final method, VEST, incorporates Retrieval-based Augmentation (RA) and Adversarial Perturbation (AP) into a stage-wise finetuning pipeline<ref:2506.04713#pg8>. The pipeline consists of three stages: (1) layer selection for PFT with validation, (2) PFT with both ID few-shot training data and the retrieved data, and (3) PFT with AP on the ID few-shot training data<ref:2506.04713#pg8>. The loss function for one-stage finetuning is L = LCE + LAP + LRA, where LAP indicates the use of AP<ref:2506.04713#pg8>.

Experimental Results

Extensive experiments on established ImageNet OOD benchmarks show that VEST significantly outperforms existing VLM adaptation methods, achieving state-of-the-art FSR performance on both ID and OOD data<ref:2506.04713#pg2>. The results demonstrate that our validation strategy effectively determines the top-k layers to PFT for better robustness and generalization<ref:2506.04713#pg9>. VEST achieves the best ID and OOD accuracy with remarkable efficiency, significantly outperforming SWAT by +2.16 and +1.02 accuracy gains on ID and OOD test data<ref:2506.04713#pg7>.

Contributions

The paper makes three key contributions: 1) We establish the robust FSR problem and comprehensively evaluate existing VLM-based FSR methods<ref:2506.04713#pg9>. 2) We propose a novel validation strategy by exploiting the few-shot ID training data and external data source, allowing hyperparameter tuning and checkpoint selection for better generalization to both ID and OOD test data<ref:2506.04713#pg9>. 3) We present a robust finetuning pipeline with our validation strategy, significantly outperforming existing VLM-based FSR methods w.r.t both ID and OOD accuracy<ref:2506.04713#pg9>.

Impacts and Limitations

Robust FSR via VLM adaptation has broad impacts on real-world applications such as data annotation, clinical diagnosis, and autonomous driving<ref:2506.04713#pg10>. However, methods that can improve OOD generalization could also lead to over-generalization<ref:2506.04713#pg10>. Limitations include the fact that RA may not work well in highly specialized tasks whose data are hard to obtain from open data resources, and the perturbed examples lack diversity<ref:2506.04713#pg10>.

Conclusions

We study robust few-shot recognition (FSR) and investigate VLM adaptation methods w.r.t test accuracy on both in-distribution (ID) and out-of-distribution (OOD) data<ref:2506.04713#pg10>. The core challenge in FSR is data scarcity, not only limited training data but also a complete lack of validation data<ref:2506.04713#pg10>. Hence, we first propose a novel and effective validation method by jointly considering the training accuracy and the accuracy on data retrieved from open data sources<ref:2506.04713#pg10>. We further leverage such open data to finetune VLM in a stage-wise manner with robust learning<ref:2506.04713#pg10>. Extensive analyses lead to our final method VEST, which significantly outperforms existing VLM adaptation methods on the ImageNet OOD benchmarks<ref:2506.04713#pg10>.

Acknowledgments

This work is supported by CK Foundation and Plato Initiative, Science and Technology Development Fund of Macau SAR (0067/2024/ITP2), University of Macau (SRG2023-00044-FST), and the Institute of Collaborative Innovation<ref:2506.04713#pg10>.

References

The paper cites numerous references, including [1] Christina Baek, Yiding Jiang, Aditi Raghunathan, and J Zico Kolter. Agreement-on-the-line: Predicting the performance of neural networks under distribution shift<ref:2506.04713#pg10>, [8] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database<ref:2506.04713#pg10>, and [52] Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs<ref:2506.04713#pg10>. The paper also references [35] Tian Liu, Huixin Zhang, Shubham Parashar, and Shu Kong. Few-shot recognition via stage-wise retrieval-augmented finetuning<ref:2506.04713#pg10>, and [42] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V. Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel HAZIZA, Francisco Massa, Alaaeldin El-Nouby, Mido Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes. DINOv2: Learning robust visual features without supervision<ref:2506.04713#pg10>. The paper also cites [54] Julio Silva-Rodriguez, Sina Hajimiri, Ismail Ben Ayed, and Jose Dolz.

1. Robust few-shot recognition (FSR) with VEST can select checkpoints that generalize well to ID and OOD test data by maximizing a harmonic mean performance gain metric, denoted as gF1t: gF1t = 2 × ∆tID × ∆tRT / (∆tID + ∆tRT).

2. The system can perform partial finetuning (PFT) of the top-k layers of the visual encoder by selecting the optimal layer count, as evidenced by the validation strategy that determines top-1 layers for PFT, which provides a good balance between ID and OOD performance.

3. The system can achieve state-of-the-art FSR performance on both ID and OOD data by utilizing Retrieval-based Augmentation (RA) in Stage 1, specifically by using the retrieved data to augment training, while simultaneously applying Adversarial Perturbation (AP) in Stage 2 to robustly finetune these layers over only the few-shot ID training data.

4. The VEST pipeline can be optimized by setting a specific perturbation magnitude of ϵ = 0.007, which is chosen by the gF1 score, leading to improved generalization across both ID and OOD test accuracies compared to vanilla PFT methods.

5. The final model can be trained using a stage-wise finetuning approach where Stage 1 involves PFT with RA, followed by Stage 2 involving PFT with AP on the ID few-shot training data, which is shown to yield better results w.r.t accuracies on ID and OOD test data compared to one-stage training.

A key challenge in Few-Shot Recognition (FSR) is the scarcity of labeled task-specific data for training. Contemporary methods mitigate this challenge by exploiting pretrained Vision-Language Models (VLMs) for their learned transferable features. Some recent ones augment the scarce data by retrieving relevant yet out-of-distribution (OOD) examples from external data sources. However, they largely overlook an equally fundamental challenge: the absence of validation data for model selection and hyperparameter tuning. As a result, the adapted VLMs overfit the few-shot training data and generalize poorly to OOD test data. We show that retrieved data provide a natural source of validation, but their direct use creates a fundamental paradox. Since the retrieved samples are inherently OOD w.r.t the few-shot ID training data, adapting a VLM to the ID data constantly degrades its performance on the retrieved data. Consequently, conventional validation relying on the accuracy of retrieved data favors no adaptation and prevents improved generalization. To resolve this dilemma, we formulate validation as a multi-objective optimization (MOO) problem between performance gains on the few-shot ID data and the performance preservation on the retrieved data. We introduce a simple yet effective validation strategy by choosing an operating point on the Pareto front of the MOO problem. This strategy enables hyperparameter tuning, which had been unrealistically done by using test data, thereby mitigating overfitting and improving generalization. We integrate it into a stage-wise finetuning pipeline, termed Validation-Enabled Stage-wise Tuning (VEST). Extensive experiments on five established benchmarks show that VEST consistently outperforms existing approaches on both ID and OOD test data.