Image Recognition with Vision and Language Embeddings of VLMs
The study evaluated language-guided and vision-only classification using dual-encoder Vision-Language Models (VLMs) on ImageNet datasets. It found that language prompts and visual similarity are complementary: some classes benefit more from text, while others are better recognized visually. A simple, learning-free fusion method combining both approaches achieved a modest accuracy improvement.