OmniCapBench: A Deep-Structured Evaluation Framework for Fine-Grained Audio-Visual Captioning

My objective is to synthesize this information into a comprehensive, detailed description that accurately reflects the framework's methodology, findings, and implications for multimodal AI evaluation.

Here is the detailed synthesis:

Comprehensive Research Summary: OmniCapBench – A Deep-Structured Evaluation Framework for Fine-Grained Audio-Visual Captioning

OmniCapBench introduces a novel deep-structured evaluation framework specifically designed to rigorously assess fine-grained audio–visual captioning models. The core innovation lies in reframing the evaluation process around atomic, verifiable audio–visual units across three distinct tracks: Entity References (R), Audio Events (E), and Visual Shots (H). This design is engineered to preserve comprehensive coverage of multimodal understanding while simultaneously enabling fine-grained error localization through deterministic constraint checks and localized Large Language Model (LLM)-based semantic comparisons.

Framework Architecture: Atomic Units and Two-Stage Scoring

The framework decomposes the evaluated caption into these native atomic units:

1. References (R): Establish persistent identities for key entities within the video content.

2. Events (E): Isolate auditory events and spoken dialogue, precisely bounded in time.

3. Shots (H): Segment the visual timeline into discrete camera shots, anchoring both references and events to specific frames.

The evaluation proceeds through a rigorous two-stage scoring pipeline:

1. Deterministic Verification: A first stage employs deterministic rules to verify the structural integrity of the output. This includes checking for valid reference IDs, well-formed time spans for events, and correct cross-modal links between modalities.

2. Localized Semantic Comparison: If the structure passes verification, aligned units are dispatched to localized LLMs to perform bounded semantic checks. This approach is critical because it eliminates the uncertainty inherent in global LLM judges and exposes specific capability deficits by isolating failure modes rather than providing a monolithic score.

OmniCapBench is instantiated across three native evaluation tracks:

• Reference Track: Catalogs persistent scenes and subjects.

• Event Track: Captures timestamped audio occurrences.

• Shot Track: Segments visual content using explicit cross-links to temporally ground references and events, often utilizing instructions like the active_events array to link current visual shots with simultaneous audio events.

Benchmark Scope and Metrics

The benchmark is extensive, comprising 786 densely annotated videos, yielding a rich set of atomic units: 5,818 entities, 6,537 audio events, and 11,419 visual shots. The evaluation suite includes specialized metrics tailored to these atomic units:

• Structural Metrics: Reference Use (RefUse), Cross-Shot Coreference Consistency (CCC), Shot/Subshot F1, and temporal Intersection over Union (tIoU) for visual elements.

• Event Metrics: Event F1 and tIoU for audio events.

• Association Metrics: Event-Shot Association (EVSA) to measure the causal relationships between audio and visual components.

Empirical Findings: Exposing Localized Failure Modes

Empirical analysis strongly suggests that global text scores often mask localized errors and can even reward structurally flawed outputs. OmniCapBench successfully demonstrates that this framework exposes critical capability deficits:

• Identity Persistence (CCC): High failure rates reveal models struggle to maintain stable entity assignments across discontinuous shots.

• Temporal Alignment: Models frequently fail to anchor events to precise timeline segments.

• Cross-modal Binding (EVSA): Models often miss the causal relationships between audio events and their supporting visual shots.

Crucially, the framework reveals that current models can describe local content but fail when asked about consistent binding across time and modality—the very failure mode that a holistic caption-quality score would tend to smooth over. The research confirms that reliable evaluation necessitates moving from monolithic text generation to verifiable atomic units, transforming performance measurement into an actionable diagnostic tool.

Methodological Rigor and Robustness

The framework's robustness is established through several key design choices:

• Decoupling Judgment: By restricting LLMs to bounded local semantic checks after deterministic rules verify structural relationships, the scoring operator becomes stable and objective, resilient to judge idiosyncrasies.

• Post-Hoc Analysis: Comparative analysis (Table B.8) shows that while post-hoc parsing can raise structural well-formedness metrics (SGC, Shot F1, Shot tIoU), it simultaneously degrades all grounding-dependent metrics (CCC, Event F1, EVSA). This joint pattern—high structural scores paired with low localized semantic scores—is precisely what deep-structured units are designed to expose.

• Actionable Diagnostics: The paper provides a clear taxonomy of failures (Table A.8) linking specific failure modes directly to corresponding rule-based metrics, providing developers with precise diagnostic information rather than vague feedback.

Broader Impact and Ethical Considerations

OmniCapBench is positioned not just as a benchmark, but as a paradigm shift in how multimodal systems are evaluated. It encourages the research community to prioritize controllable, structured video generation over opaque free-form prose generation by enforcing strict JSON schemas (Listing B.1). The broader impact is providing developers with a highly auditable, traceable diagnostic tool, ensuring that future improvements in Multimodal Large Language Models (MLLMs) represent genuine audio-visual reasoning rather than mere linguistic fluency.

Ethically, the benchmark addresses privacy and copyright concerns by curating videos from publicly accessible media platforms and distributing structured annotations alongside deterministic download scripts to original public URLs, respecting intellectual property rights while filtering out harmful or sensitive content. The dataset utilizes a diverse array of models (including Gemini 3.1-Pro, Qwen2.5-Omni, MiMo V2.5, etc.) to test the framework's efficacy across different architectures and optimization goals.

In conclusion, OmniCapBench provides a mathematically guaranteed method for measuring structured audio-visual understanding by enforcing fidelity to atomic units. It moves the industry beyond superficial text similarity scores toward verifiable diagnostic metrics that pinpoint exactly where and how multimodal reasoning fails.

1. Bold Header: Deep-Structured Evaluation for Fine-Grained Diagnostics

This framework allows AI systems to be evaluated not by opaque, whole-caption scores but by native atomic units across three tracks: References, Shots, and Events. This ensures that evaluation is deterministic constraint checks and allows for a fine-grained, reliable diagnosis, moving beyond the limitations of global LLM judges which are prone to instability.

2. Bold Header: Isolation of Modality-Specific Failures

By decoupling evaluation into deterministic rules (structural verification) and bounded semantic checks (localized LLMs), the system can pinpoint specific capability deficits. For example, this isolates Identity Drift as a distinct failure mode, where the paper shows that CCC [Cross-Shot Coreference Consistency] measures whether a model maintains a stable identity assignment for the same entity across multiple shots.

3. Bold Header: Enhanced Cross-Modal Reasoning Verification

The system can verify complex causal relationships between audio and visual streams through explicit event-shot edges. This enables the AI to detect specific failures like A-V Misalignment, where the framework checks whether each audio event is linked to the visual shot or shots that support its source, which is a failure mode often missed by holistic scores.

4. Bold Header: Elimination of Score Instability

The system achieves stable and objective scoring by restricting LLMs to perform bounded semantic comparisons on structurally aligned units, thereby eliminating the compounding uncertainty of unconstrained text-level judgments. This prevents models from being unfairly penalized for stylistic variations or hallucinated prose that might otherwise inflate a global score.

5. Bold Header: Robustness Against Specialized Fine-Tuning

The framework exposes failures in caption-specialized SFT models by testing them against the atomic unit requirement. This reveals Schema rejection and Incomplete graph construction in models like ASID-Caption, demonstrating that specialized linguistic alignment does not substitute for the controllable, graph-structured generation required by OmniCapBench.

Multimodal large language models (MLLMs) are rapidly evolving toward continuous audio--visual reasoning, creating an urgent need for evaluations that expose their capability limits. Audio--visual captioning is an ideal diagnostic task, yet current benchmarks face a coupled trade-off: whole-caption scores provide coverage without localization, local probes provide localization without coverage, and unconstrained LLM judges introduce instability. We introduce OmniCapBench (Omni-Video Caption Benchmark), a benchmark that reframes audio--visual caption evaluation as a deep-structured diagnostic framework. OmniCapBench shifts the prediction target from free-form text to sets of atomic, verifiable evaluation units across three tracks: entity references, visual shots, and audio events, enabling reliable scoring with deterministic constraint checks and localized LLM-based semantic comparisons. With 786 densely annotated videos, OmniCapBench effectively distinguishes MLLM perception errors, including temporal grounding failures, identity drift, cross-modal misalignment, and hallucinated descriptions. Evaluating frontier MLLMs reveals strong local perception but weak long-horizon audio--visual reasoning, particularly in identity drift and cross-modal misalignment, providing a fine-grained roadmap for omnimodal development.