PragMatch: Separating Pragmatic Incongruity from Cross-Modal Mismatch in Large Vision-Language Models

PragMatch introduces a controlled benchmark of 3,000 image–text pairs derived from MMSD2.0, designed to isolate pragmatic incongruity from non-pragmatic mismatch and literal congruity in multimodal sarcasm detection. Each source image is paired with three matched captions that hold visual content fixed while varying only the communicative relation: Pragmatic Incongruity (P), Non-Pragmatic Mismatch (M), and Literal Congruity (L). The mismatched captions are hard negatives retrieved from visually similar images using CLIP and filtered with a relational similarity model (RelSim) to ensure surface-level plausibility. The benchmark includes 1,000 examples per condition.

The paper evaluates four open-source vision–language models—InternVL2.5-8B, Qwen2.5-VL-7B-Instruct, LLaVA-v1.6-7B, and Idefics2-8B—under zero-shot and chain-of-thought prompting. Standard accuracy and F1 on the balanced P versus M comparison show all models exceed chance, with LLaVA-v1.6-7B reaching 74.2% accuracy and 70.8 F1. However, per-condition accuracy reveals that no model is above chance on both P and M simultaneously. InternVL2.5-8B, Qwen2.5-VL-7B, and Idefics2-8B score highly on M (96–99%) but poorly on P (8–14%), consistently defaulting to negative answers. LLaVA-v1.6-7B shows the opposite tendency, scoring 62.5% on P but only 43.1% on L, treating cross-modal inconsistency as sarcasm regardless of communicative intent.

Paired accuracy (requiring both P and M correct for the same image) falls below the 25% chance level for three of four models under zero-shot prompting (7.4%, 13.4%, and 10.2%), despite these models exceeding 96% accuracy on M in isolation. Grouped accuracy (requiring all three captions correct) never exceeds 26.4% across all eight configurations, with three at or below the 12.5% chance level. LLaVA-v1.6-7B shows the largest gap between standard accuracy and grouped accuracy, falling from 74.2% to 19.0%, a drop exceeding the 37.5 points expected of a model with no relational information.

Score-level analysis using AUROC separates representation quality from decision bias. Qwen2.5-VL-7B separates P from M well under zero-shot prompting (AUROC 90.0) despite only 13.4% paired accuracy, indicating the score carries information its decision ignores. InternVL2.5-8B's P–M AUROC (66.9) sits close to chance, pointing to a weaker representation. P–L separation exceeds P–M separation for several models (e.g., InternVL2.5-8B CoT: 83.7 vs. 63.1), showing that detecting disagreement is easier than identifying its pragmatic cause. Idefics2-8B's P–L AUROC (14.2) falls below chance, ranking literal pairs as more sarcastic than sarcastic ones. CoT prompting lowers AUROC for some models (Qwen2.5-VL-7B: 90.0 to 73.0; LLaVA-v1.6-7B: 85.5 to 74.8), showing that CoT changes the score distribution rather than only the decision boundary.

Shortcut diagnostics use three cue families—lexical, style, and OCR—with masking (removing cues from sarcastic examples) and injection (adding misleading cues to non-sarcastic examples), both preserving the underlying image–caption relation and gold label. Injection results show models are not evaluating the image–caption relation: LLaVA-v1.6-7B's accuracy on M falls from 85.9 to as low as 9.4 after OCR injection. InternVL2.5-8B and Idefics2-8B appear more stable under injection (89.8–96.3 and 94.8–96.3 against baselines of 98.0 and 96.7), but this stability reflects their tendency to predict not sarcastic for nearly every input. Masking results confirm shortcut reliance: removing cues from sarcastic condition P reduces accuracy to 4.1–6.9, showing correct predictions relied on cues rather than the image–caption relation. Qwen2.5-VL-7B is the only model whose behavior is not fully explained by this pattern, showing little change on P after masking (8.7 to 21.1 against a 13.9 baseline) but substantial drops on M after injection (76.1 to 88.3 against a 99.3 baseline).

The joint metric P−–M+ (requiring correct classification under both masking and injection simultaneously) is the strictest test, with all models scoring below their clean P–M paired accuracy and most falling well below 30, showing that relational discrimination is not robust to manipulable surface cues.

Modality ablation removes the image while keeping the caption unchanged. Except for Idefics2-8B, all models perform better with text-only inputs, suggesting predictions are driven more by textual cues than by image information. This mirrors caption-only solvability reported for related benchmarks.

Quality control includes a text-only BERT classifier achieving 99.6% item accuracy and 99.3% P–M paired accuracy on captions alone, revealing caption-level biases models may exploit. A formatting-only baseline achieves 31.3% item accuracy and 5.3% paired accuracy. Human performance on a 400-instance subset reached 85.8% item accuracy and 71.5% paired accuracy, with Cohen's κ = 0.618 for sarcasm annotation.

Qualitative examples reveal distinct failure modes: models treat injected OCR cues as genuine pragmatic evidence (e.g., Yeah, right. interpreted as speaker skepticism), style cues cause models to infer unsupported context (e.g., new speaker intentions), and lexical cues act as decision signals rather than evidence (e.g., Qwen-CoT identifies the sarcastic mechanism at baseline but predicts "No until lol" is appended).

The paper concludes that current baseline accuracy in multimodal sarcasm understanding can overestimate genuine relational understanding beyond random-chance performance. CoT prompting improves baseline sarcasm detection but also increases false positive predictions by labeling more non-pragmatic samples as sarcastic. The findings recommend complementing standard evaluation with measures of relational consistency, shortcut robustness, and visual grounding.

Improvements to AI systems:

1. Add relational consistency evaluation. Train or fine-tune models to jointly classify multiple captions for the same image (P, M, L) and require all three to be correct, rather than optimizing per-item accuracy. This forces the model to learn the communicative relation (pragmatic vs. non-pragmatic mismatch) instead of surface-level cues. The improved system can distinguish sarcasm from mere cross-modal inconsistency, reducing false positives on literal-but-mismatched pairs.

2. Implement score-level calibration with AUROC-based training. Use the score distribution (not just the decision threshold) as a training signal, as Qwen2.5-VL-7B showed high AUROC (90.0) despite poor paired accuracy. Fine-tune with a loss that maximizes separation between P and M scores (e.g., contrastive or ranking loss) while also calibrating the decision boundary. The improved system can retain relational information in its representations even when its default decision policy is biased, enabling better downstream use of confidence scores.

3. Add shortcut-robustness training via masking and injection. During training, randomly mask lexical, style, and OCR cues from sarcastic examples and inject misleading cues into non-sarcastic examples, while preserving the gold label. Use these augmented examples to penalize reliance on surface cues. The improved system can maintain high accuracy on P and M even when cues are manipulated, avoiding the failure mode where accuracy on M drops from 85.9% to 9.4% after OCR injection.

4. Enforce visual grounding through modality-ablation consistency. Train with a regularizer that penalizes performance differences between full image–text inputs and text-only inputs. This encourages the model to use image information for sarcasm detection rather than relying solely on caption text. The improved system can correctly identify sarcasm when the image provides critical pragmatic context (e.g., visual irony), and its predictions become less driven by caption-only biases (which a BERT classifier can exploit at 99.6% accuracy).

5. Implement a two-stage decision policy with relational verification. After initial classification, run a verification step that compares the current image–caption pair against a retrieved visually similar image with a known literal caption. If the model’s confidence is high but the relational score (from a separate head) is low, flip the prediction to “not sarcastic.” The improved system can avoid the default-negative bias seen in InternVL2.5-8B (8–14% on P) and the default-positive bias in LLaVA-v1.6-7B (43.1% on L), achieving above-chance performance on both P and M simultaneously.

6. Use CoT prompting with explicit pragmatic reasoning constraints. Modify chain-of-thought prompts to require the model to first identify whether the image–caption relation is literal, mismatched, or pragmatically incongruent, and only then decide sarcasm. Also penalize CoT outputs that infer unsupported context (e.g., new speaker intentions from style cues). The improved system can reduce CoT-induced false positives (e.g., Qwen2.5-VL-7B AUROC dropping from 90.0 to 73.0) while preserving genuine sarcasm detection.

7. Add a cue-attribution module. Train a small head to predict which cue family (lexical, style, OCR) is present in the input and whether it is misleading. Use this to gate the main classifier’s reliance on that cue. The improved system can resist injected OCR cues (e.g., “Yeah, right.”) by recognizing them as surface-level rather than pragmatic evidence, and can maintain accuracy on M above 90% even under adversarial cue injection.

8. Incorporate human-annotated relational labels into the loss function. Use the benchmark’s paired and grouped accuracy targets as auxiliary objectives, with a loss term that rewards correct joint classification of P, M, and L for the same image. The improved system can learn to align its internal representation with human pragmatic judgments (85.8% item accuracy, 71.5% paired accuracy), moving beyond chance-level grouped performance (max 26.4%).

Large Vision-Language Models (LVLMs) have demonstrated strong performance on multimodal benchmarks, yet it remains unclear whether they genuinely reason about relationships between images and text or rely on superficial correlations, known as shortcut learning. This question is particularly important for multimodal sarcasm detection, where successful prediction depends on recognizing pragmatic incongruity rather than treating sarcasm as simple image-text mismatch. We introduce PragMatch, a controlled benchmark of 3,000 image-text pairs derived from MMSD2.0, including original sarcastic examples and constructed literal and hard-negative pairs. We identify influential shortcut cues through systematic masking and evaluate their impact through targeted injection experiments. Our results show that LVLM predictions are sensitive to lexical, OCR-derived and stylistic cues, with injected surface signals causing substantial changes in model predictions despite unchanged underlying image-text relationships. Our findings reveal limitations in current LVLMs while PragMatch provides a systematic testbed for evaluating multimodal pragmatic reasoning beyond surface-level image-text alignment.