STEMTOX: From Collaborative Tags to Fine-Grained Toxic Meme Detection via Entropy-Guided Multi-Task Learning
cs.CV, cs.CL
Submitted: 2025-08-06
Updated: 2026-07-22
Journal ref: Transactions of the Association for Computational Linguistics, Volume 14, pages 2281-2300, 2026
DOI: 10.1162/TACL.a.801
Code: https://github.com/hate-alert/STE
License: http://creativecommons.org/licenses/by/4.0/
The gist: Memes, as a widely used mode of online communication, often serve as vehicles for spreading harmful content.
Terminology
Abstract
Memes, as a widely used mode of online communication, often serve as vehicles for spreading harmful content. However, limitations in data accessibility and the high costs of dataset curation hinder the development of robust meme moderation systems. To address this challenge, in this work, we introduce a first-of-its-kind dataset - TOXICTAGS consisting of 6,300 real-world meme-based posts annotated in two stages: (i) binary classification into toxic and normal, and (ii) fine-grained labelling of toxic memes as hateful, dangerous, or offensive. A key feature of this dataset is that it includes collaborative tags associated with the original posts, enhancing the context of each meme. In addition, we propose a novel entropy-guided multi-tasking framework -- STEMTOX -- that leverages these collaborative tags alongside visual and textual inputs within a robust classification framework. Experimental results show that incorporating these tags substantially enhances the performance of state-of-the-art VLMs in toxicity detection tasks. Our contributions offer a novel and scalable foundation for improved content moderation in multimodal online environments. Warning: Contains potentially toxic contents.
Sources
- Exploring Large Language Models for Multi-Modal Out-of-Distribution Detection
- Bridging Modalities: Enhancing Cross-Modality Hate Speech Detection with Few-Shot In-Context Learning
- Vision-Language Models Create Cross-Modal Task Representations
Related papers
- Loss Knows Best: Detecting Annotation Errors in Videos via Loss Trajectories
- AnchorWeave: World-Consistent Video Generation with Retrieved Local Spatial Memories
- Benchmarking the Robustness of Foundation Models for Mammography under Domain Shift
- MambaX-Net: Dual-Input Mamba-Enhanced Cross-Attention Network for Longitudinal MRI Segmentation
- TeleOCR: Navigating Document Parsing Across Digital and Camera-Captured Documents
- A Survey on Efficient Vision-Language-Action Models