Mitigating Multimodal LLMs Hallucinations via Relevance Propagation at Inference Time
The framework LIME addresses multimodal LLM hallucinations by enhancing perceptual input usage during inference without retraining. It uses Layer-wise Relevance Propagation (LRP) to identify that text tokens dominate generation, causing modality underutilization. LIME then applies a training-free, relevance-based objective to update the model's keyvalue representations at each decoding step, boosting reliance on visual or audio inputs and improving grounding.