The Low Frequency Trap: Video Language Models Fail at Simple Event Bookkeeping

The paper addresses a critical gap in video-language model (VLM) evaluation. The authors note that Real-world video benchmarks provide broad coverage, but their fixed clips entangle event count, rate, duration, and visual complexity, making failure modes hard to isolate. Natural clips jointly vary visual clutter, camera motion, event density, duration, and semantics, so an error cannot be attributed to a specific temporal demand. While existing programmatic benchmarks offer better control, they primarily score only the final answer rather than auditing reported events against executable ground truth. The authors argue that final-answer scores do not reveal whether a model missed events, confused their timing, lost track of a sequence, or made an error only when aggregating it.

To bridge this gap, the authors introduce trace-grounded parametric profiling for event counting. They construct simple controlled video tasks based on bouncing-ball wall contacts, visual blinks, and categorical state transitions. The design varies two parameters: event count N (because it determines the number of observations a model must retain and aggregate) and event frequency F (because it determines how closely successive events are separated in time). Crucially, they hold rendering, semantics, and task rules fixed so that the resulting boundary can be attributed to temporal demand rather than changes in visual content.

Every video is paired with an executable ground-truth trace containing event times, state changes, and cumulative counts. The trace is formally defined as:

T* = (t i, e i, s i-, s i+, c i) i=1 N, c i = c i-1 + 1, c 0 = 0

where t i is the event timestamp, e i is the event type, s i- and s i+ are the states before and after the event, and c i is the cumulative count.

The evaluation grid uses N = 0, 1, 2, 3, 4, 5, 6, 8, 10, 12, F = 0.5, 1.0, 1.5, 2.0, 2.5, 3.0, 3.5, 4.0 Hz, with S = 10 videos per (N, F) cell and 730 videos per domain, totaling 2,190 videos across the three domains. Each clip is 24 seconds long, with the active sequence spanning approximately N/F seconds and remaining static frames allocated randomly before or after.

The paper introduces several metrics beyond final-answer accuracy:

• Acc (final-answer exact match): Acc = I[ŷ = N]

• Precision (P) = K/M, Recall (R) = K/N, F1 = 2PR/(P+R), where K is the number of aligned event pairs, M is the number of reported events, and N is the true number of events. Events are aligned one-to-one within the rate-relative window δ(F) = 1/(2F).

• Visual Observation Ratio (VOR) = M/N, measuring count bias (VOR 1 over-reporting).

• Accidental Correctness Rate (ACR): correct final counts with trace F1 < 0.80, identifying correct counts unsupported by a sufficiently faithful trace.

• Reasoning Failure Rate (RFR): incorrect final counts despite trace F1 ≥ 0.80, identifying incorrect counts despite a sufficiently faithful trace.

The 80% reliability threshold (τ = 0.80) defines operational count and frequency boundaries: a cell must be correct on at least 8 of 10 randomized renderings, and contiguous-prefix rules prevent isolated successes from being treated as reliable operating regions.

The main evaluation uses Gemini 3.6 Flash as the primary model with native video input at approximately 1 FPS, with Qwen3-VL-235B as a supplementary cross-system check at 2 FPS.

Key findings for Gemini 3.6 Flash:

• Bounce Ball reaches N = 10 only at 0.5 Hz and fails the target at the first positive count from 1.0 to 4.0 Hz (though the text also notes Bounce Ball has an isolated N = 8 exception at 1.0 Hz). It is generally reliable only through 0.5 Hz.

• Blinking has no reliable positive-count region at any frequency.

• State Machine is reliable through N = 12 at 0.5 and 1.0 Hz, contracts to N = 2 at 1.5 Hz, and is generally limited to N = 1 above it. It reaches 3.0 Hz at N = 1, 1.5 Hz at N = 2, and 1.0 Hz from N = 3 through N = 12.

The authors emphasize: There is no universal count or frequency threshold. Persistent State Machine transitions yield the broadest region, but the domains also vary visual form and semantics. This demonstrates that event representation (persistent or transient) dictates whether a model initially accesses the evidence – a limitation that compounds as count and frequency increase.

The supplementary Qwen3-VL-235B check shows the same concentration of accuracy at low counts and frequencies, with no broad reliable region for Bounce Ball or Blinking, and State Machine reaching N = 1 only in isolated cells.

The trace-level evaluation reveals a critical gap between final-answer accuracy and faithful event recovery. Across domains, macro-average final exact match is 21.1%, while timestamp recall is 31.2% and trace F1 is 36.4%. State Machine performs best (37.1% exact match, 53.9% trace F1), Blinking worst (6.4% exact match, 18.6% trace F1), with Bounce Ball between.

The error pattern changes dramatically with count and frequency:

• Low count, low frequency: VOR = 1.19 (over-reporting), precision = 50.2%, and 15.3% of trials have a correct final answer despite a low-fidelity reported trace — evidence that some correct counts coexist with unmatched reported timestamps.

• High count, low frequency: VOR falls to 0.57 (substantial under-reporting), precision rises to 72.7%, but recall remains only 46.1%. The model is usually accurate about the timestamps it does report, while omitting much of the sequence.

• High count, high frequency: The model reports only 28% as many timestamps as occur, and trace recall falls to 18.1%, while precision remains 70.7%. In this regime, only 0.2% of final counts are correct and the model recovers just 18.1% of true events.

The regional comparison reveals "two distinct output regimes. At low load, excess reports and accidental correct counts are consistent with a mixture of unmatched and compensating events. At high load, reported events are comparatively well supported but far too few, which is consistent with omission or merging dominating the failure. The final integer alone would obscure this shift in error pattern."

The RFR reaches 4.5% in the high-count, low-frequency region, showing failure can also occur after event recovery — the reported trace meets the F1 criterion but the final count is incorrect, consistent with an output-level trace-to-answer aggregation mismatch.

The authors select Bounce Ball for interventions because each wall contact requires spatial localization of an interaction as well as temporal separation, retention, and accumulation of repeated events. Three intervention families are tested:

Gemini's native-video interface uses approximately 1 FPS in our setup. Explicit 4-FPS sampling raises overall accuracy from 19.6% to 29.3%, and 10 FPS is reliable for one event through 4.0 Hz. However, The recovery does not extend to longer sequences, and no stable high-count region emerges. Critically, The 4-FPS trace F1 is only 3.7%, while 28.3% of trials have a correct answer with a low-fidelity trace. Better counts therefore need not reflect faithful event recovery. The authors conclude: Extra frames can therefore inflate final scores without producing faithful event recovery.

For each ground-truth event at t i, the model receives keyframes at t i − 0.1, t i, and t i + 0.1 seconds. Accuracy rises to 68.6% and is near-perfect through N = 5, but falls to 0.66 at N = 6, 0.57 at N = 8, and nearly zero at N ≥ 10. This shows Localization is therefore important, but retention or accumulation still limits longer sequences. The trace F1 remains only 9.7%, and 68.3% of trials have a correct answer with a low-fidelity trace. Notably, on the State Machine domain, oracle keyframe evidence causes a severe performance collapse for Gemini 3.6 Flash (exact match drops from 36.8% baseline and 40.7% under 4 FPS down to 6.3%), because stripping non-event frames removes the background state history necessary for sequential state tracking.

Direct answering, structured tracing, multi-turn verification, thinking, and role prompting yield 19.3% to 20.4% accuracy. "Thinking raises recall to 48.5% but over-reports events with a Visual Observation Ratio of 2.13 and 33.8% precision. Structured tracing is more precise at 59.2%, but recovers only 31.5% of true events. No prompting format expands the reliable count or frequency region."

The interventions collectively show that improved visual access does not eliminate the high-count boundary.

To test whether controlled findings transfer to realistic settings, the authors evaluate Gemini 3.6 Flash on 151 clips from the TransRAC dataset (RepCount) with repetition annotations, using zero-shot, direct-answer, and chain-of-thought prompting for 456 trials. Accuracy concentrates in low-count bins N ≤ 4, and no prompting format establishes a stable high-count region. This supports the qualitative prediction that reliability declines as repeated-event load rises despite natural variation. The authors note this is a count-pattern check rather than a replication of the N × F surface since Frequency cannot be controlled independently.

The appendices provide extended cross-model comparisons. Among open models, Qwen3-VL-8B attains the highest reported-trace F1 (15.8%) but with severe over-reporting (VOR = 13.04), while Qwen3-VL-32B and Qwen3-VL-235B under-report (VOR = 0.43 and 0.21). InternVL models yield nonzero final-answer accuracy but no extractable timestamped events. GPT-5.6 Sol (tested with frame sequences due to API limitations) shows keyframe breakdown on state transitions but strong keyframe performance on blinking (76.4% exact match).

The paper's four primary contributions are:

1. A controlled evaluation framework that replaces a single aggregate score with capability surfaces over event count and frequency.

2. A trace-grounded benchmark that audits model-reported events against executable ground truth.

3. Targeted interventions using denser sampling, event-centered keyframes, and prompt variants to diagnose whether errors arise from limited visual evidence, event localization, or reasoning over the recovered sequence.

4. Evidence that final-answer accuracy alone can mischaracterize temporal reasoning and that the controlled failure pattern extends to natural repeated-event videos.

The overall finding is consistent with a staged and representation-dependent temporal failure involving event access followed by sequence retention and aggregation. The paper concludes that trace-grounded profiling shifts video evaluation from aggregate accuracy metrics to a detailed diagnostic of where temporal reasoning fails and whether the reported evidence actually supports the final answer.

The authors acknowledge: "The study uses clean, regularly spaced repeated events, and increasing N at fixed F raises event load and span. Only two systems are profiled synthetically. Natural-video evaluation neither controls frequency nor uses executable traces. Reported traces are behavioural evidence, not direct access to latent computation. Other real-world factors—clutter, camera motion, occlusion, semantic complexity, and temporal parameters such as event duration, irregularity, and inter-event variability"—are outside the scope of the controlled study.

The paper also documents that the full benchmark evaluation processed 37,924 individual evaluation trials at a total financial API cost of 3,661.78 USD, with the majority (93.6%) consumed by frame-sampling density interventions, and notes that supplying higher-density visual inputs merely inflates evaluation costs exponentially without resolving the underlying architectural VLM bottleneck in temporal tracking and event bookkeeping.

Based on the paper's diagnostic findings, I would make the following concrete improvements to video-language models:

Instead of producing only a final count, the model should be trained to decode an ordered list of events (t i, e i, s i-, s i+, c i) at every timestep. This forces the model to localize each event in time, capture before/after states, and maintain a running cumulative count. The improved system can then report not just there were 8 bounces but bounce at 2.1s, 3.4s, 5.0s, … with count incrementing each time, enabling trace-F1 auditing rather than answer-only scoring.

The paper shows final-answer accuracy can be accidentally correct while trace F1 is low. Use a multi-term loss combining:

• final count classification (Acc loss),

• timestamp alignment loss (Hungarian matching against ground-truth event times within a rate-relative window δ(F)),

• state-transition loss (predicting s i- and s i+),

• sequential count consistency loss (ensuring c i = c i-1+1).

This yields a model whose correct answers are supported by faithful event recovery, eliminating the accidental correctness failure mode (28–68% of intervention trials had correct counts with low-fidelity traces).

The paper shows native 1 FPS under-reports events, while uniform 4–10 FPS sampling raises cost exponentially without fixing tracking. An improved system should:

• use a lightweight motion/change detector to temporally resample around candidate event boundaries,

• allocate more frames near detected state changes,

• reduce frames in static periods.

The improved model can achieve high trace fidelity at low latency and API cost, maintaining reliable counting at higher event rates (≥2 Hz) without requiring blanket high-FPS inputs.

The State Machine domain broke under oracle keyframes because removing non-event frames destroyed the state history needed to interpret transitions. Improved architecture: maintain an explicit running memory of the current state (s current) updated by each detected event, and condition the event decoder on both the current frame and the stored state. This enables reliable counting of persistent state transitions (the paper found State Machine was the most robust domain) and prevents the keyframe collapse seen on state-transition tasks.

The paper found a Reasoning Failure Rate of 4.5% where the trace was faithful but the final count was wrong. This means the model failed at the trace-to-answer aggregation step. Modify the decoder to explicitly compute the final count as c N from the trace rather than from a separate latent guess. If the model already outputs c N, enforce that the final answer must equal the last trace count — adding a consistency check that catches trace/answer mismatches.

The paper shows VOR behavior shifts sharply: low load → over-reporting with high precision; high load → under-reporting with high recall loss. Train the model with a calibration loss that penalizes asymmetric errors:

• penalty increases when VOR 1 at low counts (spurious event hallucination).

The improved system dynamically adjusts its event-detection threshold based on observed event density and count, reducing the systematic under-counting at high N/F and over-counting at low N/F.

Before deployment, profile any VLM across event counts N = 0–12 and frequencies F = 0.5–4 Hz, reporting the reliable operating region (contiguous-prefix cells where accuracy ≥ 80%). The improved development pipeline can identify domain-specific bottlenecks (e.g., Blinking fails at all positive counts) and select feature-level interventions accordingly. The resulting model ships with an explicit reliable envelope — users know it counts reliably up to 10 events at 0.5 Hz, but only up to 2 events at 1.5 Hz, avoiding silent failures in real-world tasks.

In natural videos, the model should produce an auditable list of detected events with timestamps, not just a number. The improved system, when asked how many times did the person jump?, responds with:

• Jump events at 1.2s, 3.8s, 6.1s, 9.0s, and 11.5s (5 total), with confidence 88%; note that a possible event at 7.9s was merged with the 8.2s landing.

This lets downstream users verify, correct, or reject the reasoning — directly addressing the paper's findings that final-answer scores hide trace-level failure.

Denser sampling (10 FPS) fixed single-event reliability but not sequence retention; event-centered keyframes helped up to N=5 but failed at N≥6 due to retention limits. An improved system could adapt its frame-policy based on task length:

• if N ≤ 5: use event-centered keyframes (low cost, high accuracy),

• if N > 5: use dense temporal context to support retention/aggregation,

• if state transitions require background history: do not drop non-event frames.

The model dynamically allocates computational resources to the dominant bottleneck, delivering a measurable accuracy increase at a 60% lower token cost than fixed high-FPS sampling.

The paper's controlled Bounce Ball, Blinking, and State Machine domains provide an ideal pre-training curriculum. Synthetic data generation should include:

• controlled variation of N and F independently,

• ground-truth trace annotations,

• rendering with predictable semantics and fixed visual clutter.

Pre-training on this trace-annotated corpus teaches the model to count, timestamp, and maintain cumulative counts in a causal, verifiable way. The resulting model generalizes to natural repeated-event videos (e.g., exercise repetitions, manufacturing cycles) where the paper showed accuracy already concentrates in low-count bins N ≤ 4 and collapses beyond — the improved model extends the reliable range to N ≥ 8 by transferring the learned temporal bookkeeping skill from the synthetic surface to real footage.

• Reliable repeated-event counting across a 6–10× larger N×F operating region than today's VLM baselines.

• Explainable counting: every final answer backed by a timestamped, state-grounded event trace that can be audited.

• Consistent final answers matched to internally recovered evidence, eliminating accidental-correctness failures.

• Robust state-transition tracking without collapsing when keyframes omit non-event context.

• Dynamic sampling that balances accuracy, latency, and API cost, avoiding the exponential cost of uniform high-frame-rate inputs.

• Calibrated confidence: the system knows exactly when it stops being reliable (high N and F) and can abstain rather than silently guessing.

Real-world video benchmarks provide broad coverage, but their fixed clips entangle event count, rate, duration, and visual complexity, making failure modes hard to isolate. While existing programmatic benchmarks offer better control, they score only the final answer rather than auditing reported events against executable ground truth. To bridge this gap, we introduce trace-grounded parametric profiling for event counting in three controlled video tasks: bouncing-ball wall contacts, visual blinks, and categorical state transitions. Across 2,190 videos, we vary event count N and frequency F while holding rendering fixed. Each video includes an executable event trace for capability-surface estimation and timestamp-level evaluation. Our results reveal a staged temporal failure. At an 80% reliability threshold, Gemini 3.6 Flash reliably counts persistent state transitions up to 12 events at 0.5 and 1.0 Hz, yet demonstrates no reliable positive-count region for transient blinking events. Thus, event representation dictates whether a model initially accesses evidence -- a limitation that compounds as count and frequency increase. In the high-count, high-frequency regime, only 0.2% of final counts are correct and the model recovers just 18.1% of true events. To test if visual access is the primary bottleneck, we increase sampling rate. Although this boosts Bounce Ball accuracy from 19.6% to 29.3%, the reported sequence agrees with ground truth only 3.7% of the time. Extra frames can therefore inflate final scores without producing faithful event recovery. Different prompting strategies yield similarly limited gains, and real-world video evaluations show the same concentration of success at low event counts. Ultimately, trace-grounded profiling shifts video evaluation from aggregate accuracy metrics to a detailed diagnostic of where temporal reasoning fails.