AI Wizards at EXIST 2026: Hierarchical Soft-Label Learning for Multimodal Sexism Identification in Memes

The gist The system presents a hierarchical multi-task learning approach for multimodal sexism identification in memes by modeling tasks as conditional soft-label predictions over empirical annotator distributions, achieving top performance on Task 2.3.

How it works

The core of the system maps fixed Gemini Embedding 2 vision-language representations through a lightweight Gated MLP backbone consisting of one SwiGLU block with an expansion factor of 2 and dropout of 0.2, which results in a trainable portion containing only 3.5M parameters The gating mechanism learns to suppress benign confounders and amplify hostile cross-modal contradictions. This shared representation h feeds three independent linear heads producing logits for each subtask.

Soft-Label Training and Dynamic Task Weighting

The system is trained against empirical soft targets using KL divergence, which penalizes the model when it places mass on categories annotators collectively rejected. To balance three heterogeneous tasks, fixed scalar weights are replaced by homoscedastic uncertainty weighting, introducing learnable variance parameters sigma i. This formulation enables self-paced learning because when task losses are high, sigma i increases to reduce that task’s contribution, and the log sigma i term prevents complete task abandonment. Furthermore, losses for Tasks 2.2 and 2.3 are masked to propagate gradients only for instances where downstream annotations exist.

Hierarchical Constraints and Architectural Design

The system enforces hierarchical constraints through conditional training and decoding. The architectural design includes a shared backbone that processes pre-trained Gemini Embedding 2 features, followed by the SwiGLU blocks, which are then routed to all classification heads. An explicit structural conditioning variant tested in Run 3 scaled the shared representation by the predicted sexism probability with stopped gradients to suppress downstream features when sexism probability is low. The comparable performance suggests that hierarchical constraints are adequately captured through conditional loss masking and probabilistic decoding alone without architectural gating.

Task Formulation and Evaluation

The research problem is structured around three hierarchically organized subtasks: Sexism Identification (Task 2.1), Source Intention Detection (Task 2.2), and Sexism Categorization (Task 2.3). Task 2.1 is a binary classification task determining whether a meme depicts a sexist situation or endorses sexist behavior. Task 2.3 is a multi-label classification task assigning each sexist meme to one or more of five categories, including IDEOLOGICAL-INEQUALITY, STEREOTYPING-DOMINANCE, OBJECTIFICATION, SEXUAL-VIOLENCE, and MISOGYNY-NON-SEXUAL-VIOLENCE. The system uses soft-label decoding for inference on Task 2.2 and Task 2.3 to output joint distributions over annotator possibilities.

Key Findings

The model achieved strong performance on the official Soft-Soft leaderboard, ranking 1st on Task 2.3 and fourth on Tasks 2.1 and 2.2. Local evaluation showed meaningful seed variance, especially for downstream tasks, with Run 2 performing best for Task 2.3. The gap between Soft-Soft and Hard-Hard rankings confirms that distribution prediction is better matched to soft-label prediction objectives. The system's probabilistic outputs retain discriminative structure sufficient for non-trivial hard-label results even with naive thresholding.

Future work should investigate open-weight embedding backbones, task-specific hard-label calibration, and deeper integration of physiological signals through non-linear fusion architectures. The authors also noted that the statistical analysis of physiological signal separability was confined to EEG data because gaze and heart-rate signals were excluded a priori based on feature dimensionality and hypothesized task relevance. The work is subject to limitations, including reliance on a proprietary model like Gemini Embedding 2, the fixed decision threshold of 0.5 for hard-label decoding, and the exclusion of certain physiological modalities from linear analysis. The authors used OpenAI-ChatGPT during preparation for grammar and spelling checks. The work is copyrighted under Creative Commons License Attribution 4.0 International (CC BY 4.0)

--- Page 1 ---

AI Wizards at EXIST 2026: Hierarchical Soft-Label Learning for Multimodal Sexism Identification in Memes

Matteo Fasulo, Antonio Gravina, Luca Tedeschini and Luca Babboni contributed equally

Abstract We present the AI Wizards submission to EXIST 2026 for multimodal sexism identification in memes The task is composed of three, increasingly harder subtasks We model them hierarchically as conditional soft-label prediction over empirical annotator distributions Our system maps fixed Gemini Embedding 2 vision-language representations through a lightweight Gated MLP trained with KL divergence and homoscedastic uncertainty weighting Our submissions ranked first on Task 2.3 and fourth on Tasks 2.1 and 2.2 on the official Soft-Soft leaderboards The code is available at https://github.com/NLP-AI-Wizards/EXIST-2026

Keywords sexism identification, multimodal memes, learning with disagreement, hierarchical classification, soft labels

1. Introduction Exist 2026 structures sexism detection as three hierarchically nested subtasks detailed in Section 3: binary sexism identification, source intention detection for sexist content and fine-grained multi-label categorization across five sexism types Critically, the task adopts the Learning with Disagreement (LeWiDi [8]) paradigm requiring systems to predict the full empirical annotator distribution rather than a single majority label We address all three subtasks as hierarchical conditional multi-task learning over pre-computed vision-language embeddings preserving semantic dependencies while directly optimizing for soft-label evaluation Our main contributions are as follows.

2. Related Work 2.1 Multimodal Sexism Detection Early detection systems focused on text-only platforms [9, 10] The introduction of multimodal benchmarks (SemEval-2022 MAMI [4] and the Hateful Memes Challenge [5]) revealed that unimodal systems fail when toxicity arises only from text-image interaction [11] The EXIST lab has tracked this evolution from text (2021–2023) to memes (2024) [12] to video (2025) [9] As sexism detection has moved into the multimodal setting it has increasingly drawn on architectures originally developed for the broader hate speech domain which we review next

3. Dataset and Problem Formulation EXIST 2026 formalizes the research problem through three hierarchically organized subtasks focused on multimodal sexist content Sexism Identification (Task 2.1) A binary classification task aimed at determining whether a meme depicts a sexist situation or endorses sexist behavior (YES) or, conversely, does not contain sexist content (NO) Source Intention Detection (Task 2.2) A binary classification task defined exclusively for memes previously labeled as sexist it distinguishes between DIRECT intention where the communicative goal is to produce inherently sexist content or to incite sexist attitudes JUDGEMENTAL intention where the content portrays sexist situations or behaviors with the explicit objective of condemning or criticizing them Sexism Categorization (Task 2.3) A multi-label classification task that assigns each sexist meme to one or more of the following five categories

4. Methods Our system rests on three principles (1) extracting rich multimodal semantics without fine-tuning massive models on limited data (2) predicting annotator disagreement distributions rather than forced labels and (3) enforcing hierarchical constraints through conditional training and decoding 4.1 Shared Backbone Let fVLM: 768 denote the frozen Gemini Embedding 2 encoder [18] producing a fixed embedding x per meme We pass x through a lightweight Gated MLP backbone consisting of one SwiGLU block with expansion factor 2 and dropout 0.2: SwiGLU(x) = (xW1)⊙ SiLU(xW2), (1) h(l+1) = h(l) + W3((SwiGLU (LN(h(l))))). (2) The gating mechanism learns to suppress benign confounders and amplify hostile cross-modal contradictions The resulting representation h feeds three independent linear heads producing logits for each subtask. The trainable portion contains only 3.5M parameters. 4.2 Soft-Label Training and Dynamic Task Weighting The Soft-Soft evaluation expects systems to predict full annotator distributions We train each head against empirical soft targets using KL divergence [20] which penalizes the model when it places mass on categories annotators collectively rejected For Task 2.

1. textbf Hierarchical Soft-Label Prediction via KL Divergence and Uncertainty Weighting. The system can model sexism detection as hierarchically nested subtasks by mapping fixed Gemini Embedding 2 representations through a lightweight Gated MLP trained with KL divergence and homoscedastic uncertainty weighting. This allows the model to predict the full empirical annotator distribution rather than a single label, specifically optimizing for Task 2.3, which ranked 1st on Task 2.3 on the official leaderboard.

2. textbf Dynamic Multi-Task Weighting for Heterogeneous Tasks. The system employs "homoscedastic uncertainty weighting [21], introducing learnable variance parameters σi: 'This formulation enables self-paced learning: when task losses are high, σi increases to reduce that task’s contribution; the log σi term prevents complete task abandonment.' This mechanism automatically balances three heterogeneous tasks—differing in label spaces, class imbalance, conditional support, and intrinsic difficulty."

3. textbf Hierarchical Decoding for Conditional Constraints. The model enforces structural dependencies through conditional loss masking and probabilistic decoding that respect the task taxonomy without architectural overhead. For example, Task 2.3 decoding uses the structure p(1)p(3)c, ensuring that a meme must have a non-zero sexism probability to proceed to fine-grained categorization.

4. textbf Explicit Structural Conditioning via Detached Soft-Gating (Ablation Study). The system can explicitly suppress irrelevant features by using hdownstream = h ⊙ sg [p(1)] with stop-gradient, which suppresses downstream features when sexism probability is low. This confirms that hierarchical constraints are adequately captured through conditional loss masking and probabilistic decoding alone, without architectural gating.

5. textbf Multimodal Semantic Extraction from Frozen Embeddings. The system achieves high performance while maintaining efficiency by using frozen Gemini Embedding 2 representations passed through a lightweight Gated MLP backbone consisting of one SwiGLU block, resulting in only 3.5M parameters. This avoids the computational cost of fine-tuning large generative models on limited data.

We present the AI Wizards submission to EXIST 2026 for multimodal sexism identification in memes. The task is composed of three, increasingly harder subtasks. We model them hierarchically as conditional soft-label prediction over empirical annotator distributions. Our system maps fixed Gemini Embedding 2 vision-language representations through a lightweight Gated MLP trained with KL divergence and homoscedastic uncertainty weighting. Our submissions ranked first on Task 2.3 and fourth on Tasks 2.1 and 2.2 on the official Soft-Soft leaderboards. The code is available at https://github.com/NLP-AI-Wizards/EXIST-2026