Half-Truths Break Similarity-Based Retrieval
CS-CLIP addresses 'half-truth vulnerabilities' in CLIP models where adding a plausible but incorrect detail boosts similarity scores. The method decomposes captions into entity and relation units, creates foil versions of these units, and trains the model to score the correct unit higher than its foil. This results in better compositional understanding and higher accuracy on benchmarks.