OmniConfess: Eliciting Token Confessions to Mitigate Omni-Modal Hallucination
cs.CL
Submitted: 2026-10-02
Updated: 2026-10-05
Code: https://github.com/RongHuiQiang/OmniConfess
License: http://creativecommons.org/licenses/by/4.0/
The gist: Omni-modal large language models (OmniLLMs) unify text, images, audio, and video, yet hallucinate when generation relies on the wrong evidence.
Terminology
Abstract
Omni-modal large language models (OmniLLMs) unify text, images, audio, and video, yet hallucinate when generation relies on the wrong evidence. Existing inference-time methods can reduce hallucinations, but rarely reveal which evidence sustains a generated commitment. We introduce OmniConfess, a training-free method for mitigating omni-modal hallucinations. It fixes a candidate response and re-scores it at token resolution under controlled channel-wise evidence interventions, producing a structured token-by-channel confession that reveals the response's evidential dependence. OmniConfess uses this confession to preserve grounded content and correct commitments driven by irrelevant or contradictory evidence. To evaluate OmniConfess, we construct OmniHalluBench, a 3,540-example benchmark built from six datasets spanning text, image, audio, and video settings and both judgment and free-form generation. Experiments show that OmniConfess mitigates hallucinations across heterogeneous modality and task settings. Our code and benchmark are publicly available at https://github.com/RongHuiQiang/OmniConfess.
Sources
- MoD-DPO: Towards Mitigating Cross-modal Hallucinations in Omni LLMs using Modality Decoupled Preference Optimization
- Where MLLMs Attend and What They Rely On: Explaining Autoregressive Token Generation
- DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Models
- MAD: Modality-Adaptive Decoding for Mitigating Cross-Modal Hallucinations in Multimodal Large Language Models
- OmniHalluc-L: Counterfactual Benchmarking and Modality-Perturbation Reliability Calibration for Long-Form Omni Hallucination
- Multi-Modal Hallucination Control by Visual Information Grounding
- HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Models
- OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation
- PubMedQA: A Dataset for Biomedical Research Question Answering
- OMHBench: Benchmarking Balanced and Grounded Omni-Modal Multi-Hop Reasoning
- Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding
- Token Activation Map to Visually Explain Multimodal LLMs
- Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning
- PhD: A ChatGPT-Prompted Visual hallucination Evaluation Dataset
- RAGTruth: A Hallucination Corpus for Developing Trustworthy Retrieval-Augmented Language Models
- Alleviating Hallucination in Large Vision-Language Models with Active Retrieval Augmentation
- Trusting Your Evidence: Hallucinate Less with Context-aware Decoding
- Aligning Large Multimodal Models with Factually Augmented RLHF
- AVHBench: A Cross-Modal Hallucination Benchmark for Audio-Visual Large Language Models
- Self-Consistency Improves Chain of Thought Reasoning in Language Models
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering