Senses Wide Shut: A Representation-Action Gap in Omnimodal LLMs
The study tested how omnimodal models handle contradictions between text and sensory input using a new benchmark. It found that internal model states reliably capture these mismatches, even when the model fails to reject false claims in its final output. This suggests the problem isn't perception itself, but rather a failure in translating those internal mismatch signals into actual rejection behavior.