FBHM: Functional Benchmarking and Steering of VLMs for Hateful Meme Detection

arXiv:2605.31349 · cs.CL, cs.AI, cs.CV, cs.MM · Submitted 2026-05-29 · Read on arXiv

cs.CL, cs.AI, cs.CV, cs.MM

Submitted: 2026-05-29

Updated: 2026-08-29

Code: https://github.com/MaartenGr/BERTopic

License: http://creativecommons.org/licenses/by/4.0/

The gist: Hateful meme detection remains a formidable challenge for vision-language models, as existing benchmarks are structurally observational - confounding rhetorical hate mechanisms with target community

Terminology

Abstract

Hateful meme detection remains a formidable challenge for vision-language models, as existing benchmarks are structurally observational - confounding rhetorical hate mechanisms with target community features and preventing causal evaluation of model vulnerabilities. To address this, we introduce FBHM, a systematically curated benchmark of Functionality Based Hateful Memes constructed along two orthogonal axes: 25 distinct rhetorical functionalities and 10 target communities (5,000 memes total). Benchmarking state-of-the-art VLMs reveals a severe generalization gap: models highly accurate on standard datasets catastrophically drop to near-random performance on FBHM, proving they exploit dataset-specific heuristics rather than robust multimodal reasoning. To efficiently close this gap, we propose LSV (learnable steering vectors), an ultra-low data regime strategy that applies a causal intervention objective on as few as 500 steering samples (50 unique base memes), boosting FBHM performance by 30 Macro-F1 points while outperforming in-context learning and PEFT without degrading source-domain performance.

Sources

Related papers