HumanScore: Benchmarking Human Motions in Generated Videos

summary

The gist

HumanScore is designed to automatically detect subtle biomechanical violations produced by AI video generators, even when they are imperceptible to the human eye The paper presents HumanScore…

In short

HumanScore is a framework to automatically detect subtle biomechanical errors in AI-generated videos. It scores motion quality using six metrics—covering anatomy, kinematics, and dynamics—to reveal gaps between visual realism and physical plausibility. Results show leading models still struggle with accurate human motion.

Key concepts

Evaluation Framework
A three-tier system for judging motion quality: base tier checks anatomical correctness (correct body structure), middle tier checks kinematic correctness (feasible poses), and the top tier checks kinetic correctness (smooth, physically plausible movement).
Metric Design Details
Calculations use per-frame outputs like bt (normal/abnormal flag) and mt (severity score) to aggregate into video metrics. The final score is a weighted sum of three components—r, s, and p—which are mapped to a 100-point scale representing quality.
Kinematic Extremes
This metric evaluates how natural the motion is by checking joint angular and body-segment linear velocities against established biomechanical standards. High scores indicate more natural and physically plausible movement patterns.
Human Preference Alignment
The HumanScore metrics strongly correlate with human preference ratings, showing high Spearman correlation coefficients. This validates the framework as a reliable way to measure how well generated human motion aligns with what humans perceive as realistic.

Terminology used across episodes

This episode discusses

The paper

HumanScore: Benchmarking Human Motions in Generated Videos · Read on arXiv

Stanford University · Peking University

Recent advances in model architectures, compute, and data scale have driven rapid progress in video generation, producing increasingly realistic content. Yet, no prior method systematically measures how faithfully these systems render human bodies and motion dynamics. In this paper, we present HumanScore, a systematic framework to evaluate the quality of human motions in AI-generated videos. HumanScore defines six interpretable metrics spanning kinematic plausibility, temporal stability, and biomechanical consistency, enabling fine-grained diagnosis beyond visual realism alone. Through carefully designed prompts, we elicit a diverse set of movements at varying intensities and evaluate videos generated by thirteen state-of-the-art models. Our analysis reveals consistent gaps between perceptual plausibility and motion biomechanical fidelity, identifies recurrent failure modes (e.g., temporal jitter, anatomically implausible poses, and motion drift), and produces robust model rankings from quantitative and physically meaningful criteria.

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "HumanScore: Benchmarking Human Motions in Generated Videos".

Jane: Here's the gist of the paper. HumanScore is designed to automatically detect subtle biomechanical violations produced by AI video generators, even when they are imperceptible to the human eye The paper presents HumanScore, a systematic framework to evaluate the quality of human motions in AI-generated videos This benchmark extracts structured human motion from generated videos and scores it using a suite of anatomically- and biomechanically grounded metrics Our analysis reveals consistent gaps between perceptual plausibility and motion biomechanical fidelity, identifies recurrent failure modes (e.g., temporal jitter, anatomically implausible poses, and motion drift), and produces robust model rankings from quantitative and physically meaningful criteria.

Tom: Evaluation Framework HumanScore defines six interpretable metrics spanning kinematic plausibility, temporal stability, and biomechanical consistency The evaluation framework follows a three-tier hierarchy: the base tier enforces anatomical correctness (valid human topology with stable body proportions), the middle tier enforces kinematic correctness (geometric feasibility of poses), and the top tier enforces kinetic correctness (temporal coherence and physically plausible dynamics).

Jane: The metrics are designed to follow this hierarchy at six evaluation dimensions: spurious limbs or inconsistent proportions, hyperextension or interpenetration, non-human spikes or jitter, joint range of motion, self-collision, and kinematic extremes.

Tom: Metric Design Details The metrics are calculated using a generic framework where per-frame calculations produce three outputs: bt – a binary indicator that flags whether a frame is normal or abnormal, mt – a relative metric score that measures how “bad” a frame is, in the range (0, 1), and Lmax – the length of the longest consecutive abnormal (according to bt) frame segment.

Paper summary: Tom: To break down the paper, they define six specific metrics that cover kinematic plausibility, temporal stability, and biomechanical consistency. They use a three-tier hierarchy where you check anatomical correctness first, then geometric feasibility for poses, and finally kinetic correctness for how the motion actually flows over time.

Jane: That hierarchy makes sense because it starts with the basics—does the body even have a stable shape to begin with?

Lu: Yes, the base tier focuses on anatomical correctness, meaning checking if you have valid human topology with stable body proportions across all frames.

Meng: And then they move up to kinematic correctness, which is checking if the poses are geometrically feasible—like can a limb actually reach that position without breaking physics?

Tom: Then there's the top tier, kinetic correctness, which is about temporal coherence and physically plausible dynamics over time. That’s where you check for things like motion drift or sudden jolts in the action.

Lalam: The metrics are calculated frame by frame using a generic system that outputs three values: bt, mt, and Lmax.

Jane: So bt is a binary indicator flagging if a frame is normal or abnormal, mt is a relative metric score from zero to one measuring how bad that frame is, and Lmax tracks the longest streak of abnormal frames.

Tom: And then they combine these into one final video-level metric using a weighted sum: D equals alpha r plus beta s plus gamma p, and the final score is one hundred minus D, with those weights being zero point five for r, zero point three for s, and zero point two for p <ref:2604.20157#pg2>.

Lu: Specifically they use metrics like Extra Limbs to detect anomalous limbs using HADM to find abnormal limbs, Bone Length by fitting OpenSim skeletons to see the relative length deviation from the median length over time.

Meng: And they also check self-collision using a bounding volume hierarchy and fast triangle-triangle intersection tests, which is pretty intense computation for frame-level checking.

Jane: They also have checks for joint range of motion against anatomical limits, kinematic extremes by looking at joint angular and body segment linear velocities against biomechanics standards, and motion smoothness by comparing angular acceleration and jerk to per degree of freedom limits.

Conclusion: Tom: So, to wrap up the "HumanScore: Benchmarking Human Motions in Generated Videos" paper, the core message is that these six metrics provide a principled, quantitative evaluation of motion realism from a biomechanical perspective.

Jane: They show that when you apply this benchmark to state-of-the-art video generators—like Sora two point two and Veo three point one fast—you find consistent weaknesses in motion fidelity despite their visual appeal.

Lu: The paper's contributions are providing the first systematic evaluation framework centered on human motion in generated videos, proposing quantitative indicators grounded in anatomy and biomechanics with transparent baselines, and applying it to show where the gaps between visual appearance and physical realism are.

Meng: What this means for deployment is that even if a model looks great visually, you still have to check these physical plausibility scores before you trust it for anything important like sports commentary or training simulations.

Lalam: The authors found that the correlation between their HumanScore metric values and human preference win ratio shows strong agreement across all six evaluation dimensions, with Spearman correlation coefficients close to one point zero.

Tom: That strong agreement is significant because it means the numbers they calculate actually track what people perceive as good motion very well.

Jane: So, in simple terms, this paper gives us a reliable tool to distinguish real human movement from synthetic movement in AI videos. It shifts the focus from just looking good to being physically believable.

More episodes

← Home