HiVE-VQA: Hierarchical Interaction via Volumetric Embedding for Video Quality Assessment

summary

The gist

HiVE-VQA proposes HiVE-VQA, the first framework to apply a JEPA-based video encoder to this task It extracts token grids from multiple depths of a pre-trained V-JEPA 2.1 encoder and stacks them into…

In short

HiVE-VQA introduces a framework that uses a pre-trained V-JEPA 2.1 encoder to create hierarchical video features. It stacks these features into a 3D volume and fuses them using a compact 3D convolutional head. This method outperforms existing state-of-the-art models in video quality assessment by effectively capturing both local and global context from the video data.

Key concepts

JEPA-based Video Encoder
This is a self-supervised model pre-trained on videos that learns rich, general spatial and temporal features. HiVE-VQA repurposes this encoder to extract high-quality representations of video content, which are then used as the foundation for quality prediction.
Hierarchical Feature Extraction
The framework extracts video features from different depths of the pre-trained encoder and organizes them into a 3D volume. This structure allows the model to analyze visual information at multiple levels—from fine details to broader temporal patterns—for a more comprehensive quality assessment.
Volumetric Quality Prediction
A lightweight 3D convolutional head is used to fuse the hierarchical feature volume across its spatial and temporal dimensions. This fusion process combines local visual context with global video context to predict a single, accurate scalar score representing the video's quality.

Terminology used across episodes

This episode discusses

The paper

HiVE-VQA: Hierarchical Interaction via Volumetric Embedding for Video Quality Assessment · Read on arXiv

University of Bristol

The proliferation of user-generated video content has made accurate and robust no-reference video quality assessment (NR-VQA) a highly important challenge. Existing research increasingly relies on architecturally complex multi-branch pipelines or computationally intensive fine-tuning of single-branch backbones. In this paper, we investigate whether a single frozen video foundation model can serve as an effective NR-VQA backbone, and propose HiVE-VQA, the first framework (to our knowledge) to apply a JEPA-based video encoder to this task. HiVE-VQA extracts token grids from multiple depths of a pre-trained V-JEPA 2.1 encoder and stacks them into a hierarchical spatio-temporal volume, which a compact 3D convolutional head fuses across its layer, spatial, and temporal axes, combining low-level distortion cues from shallow layers with the semantic and temporal context captured at greater depths. The encoder remains frozen throughout, so only the head is trained: 5.2M parameters (5.7% of the total), with no auxiliary multi-modal models and an inference cost that is invariant to video resolution. Trained solely on LSVQ and evaluated without any fine-tuning, HiVE-VQA outperforms 17 state-of-the-art (SOTA) methods in SROCC and PLCC on both intra-dataset (LSVQ-test, LSVQ-1080p) and cross-dataset (KoNViD-1k, LIVE-VQC) benchmarks. These results indicate that hierarchical, depth-wise fusion of features from a single frozen video foundation model provides a more effective and efficient basis for NR-VQA than task-specific multi-network fusion. Code and model checkpoints are available at https://github.com/RobHam99/HiVE-VQA.

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "HiVE-VQA: Hierarchical Interaction via Volumetric Embedding for Video Quality Assessment".

Jane: Here's the gist of the paper. HiVE-VQA proposes HiVE-VQA, the first framework to apply a JEPA-based video encoder to this task It extracts token grids from multiple depths of a pre-trained V-JEPA 2.1 encoder and stacks them into a hierarchical spatio-temporal volume, which a compact 3D convolutional head fuses across its layer, spatial, and temporal axes This approach demonstrates that hierarchical, depth-wise fusion of features from a single frozen video foundation model provides a more effective and efficient basis for NR-VQA than task-specific multinetwork fusion HiVE-VQA outperforms 17 stateof-the-art (SOTA) methods in SROCC and PLCC on both intra-dataset (LSVQ-test, LSVQ-1080p) and cross-dataset (KoNViD-1k, LIVE-VQC) benchmarks Framework Overview The proposed framework repurposes a self-supervised video encoder, pre-trained with the V-JEPA 2.1 objective

30: , as a hierarchical feature extractor for no-reference video quality assessment The framework consists of three stages: (i) Data sampling, which maps an input sequence X to a fixed-size fragment collage X' using the fragment sampling strategy introduced in

45: , (ii) Hierarchical feature extraction, where features are extracted from a uniformly spaced subset of L transformer blocks and folded into spatio-temporal grids to create a hierarchical feature volume V, and (iii) Volumetric quality prediction, where a lightweight 3D convolutional head fuses the hierarchical volume across its layer, spatial, and temporal axes to predict a single scalar quality score sˆ.

Tom: Training Signal Construction The model is trained using a hybrid loss function configured to optimize both the ranking monotonicity and linear correlation with human perceptual quality This loss is defined as L hybrid = ω rank L rank + ω PLCC L PLCC.

Jane: The ranking-based loss function, L rank, is designed to guide the model to learn the quality monotonicity of usergenerated content by defining s ij = sign(y i - y j) and prediction differences d ij = hat y i - hat y j.

Tom: The Differentiable Pearson Linear Correlation Coefficient (PLCC) losses are common in the quality assessment field, containing two terms: L MSE which minimizes the normalized mean squared error and L corr which aligns the prediction scale with the correlation coefficient.

Paper summary: Lu: The authors of HiVE-VQA are showing that this hierarchical feature extraction approach using the VJEPA two point one encoder can effectively represent low level quality information, and their proposed lightweight quality module captures both local and global context from a hierarchical feature volume <ref:2611.17146#pg1>.

Tom: It’s worth remembering that they achieved these results by repurposing an existing self supervised video encoder pre trained with the VJEPA two point one objective to serve as a hierarchical feature extractor for no reference video quality assessment <ref:2611.17146#pg1,no reference video quality assessment>.

Jane: So, in simple terms, this paper is about taking a powerful tool that already understands video very well and structuring its outputs in a layered way so you can assess how bad the quality is without needing perfect examples of good or bad videos.

Meng: It makes sense that they focused on fusing those features across layer, spatial, and temporal axes because they needed to capture both the immediate visual glitches and the broader context of what’s happening in the video.

Lalam: The main implication is that we don't always need entirely new architectures for every quality assessment problem; sometimes structuring existing powerful models hierarchically is enough to get accurate prediction, especially for user generated content.

Tom: They did show that this method achieves strong generalizability across both cross dataset and task specific evaluations, meaning it holds up well even when you move from one type of video quality test to another.

Jane: So, what we’re seeing here is a framework that uses existing powerful tools to build something specialized for no reference video quality assessment, which is a pretty significant direction for the field.

Conclusion: Tom: So we've been looking at HiVE-VQA, and now we’re getting to the conclusion of this paper titled "HiVE-VQA: Hierarchical Interaction via Volumetric Embedding for Video Quality Assessment."

Jane: Right, so what is the big picture here? Basically, the authors took an existing video encoder from a self supervised learning model and they layered its features up into a three dee volume to see if that helps us score video quality without needing any human labels.

Lu: The core idea they are pushing is that this hierarchical fusion of features across layer, spatial, and temporal axes provides a much better basis for no reference video quality assessment than just using a single feature vector from the final part of the encoder.

Meng: From my side, I’m thinking about deployment. If we can use this structure to get results like those reported on SROCC and PLCC, that means we might be able to run quality checks much faster than those other complex models you mentioned earlier.

Tom: Exactly, because they show that at 1080p resolution, this HiVE-VQA is about eleven times less computationally expensive than some of the state of the art methods we've seen.

Jane: That’s a huge number to drop on the listener—eleven times less computation means it could be practical for real time applications or high volume processing.

Lu: And they show that even when they fine tune this on user generated content, it still improves performance over the best multi branch model by about one point one percent in both ranking and correlation scores.

Tom: One point one percent is small, but when you compare it to the gains seen on pre training data, like the LSVQ test where they improved SROCC by a point seven four percent, that's meaningful.

Jane: It means this approach isn't just theoretical; it actually translates into measurable accuracy across different testing environments.

Meng: But what about the limitations? The authors mention focusing on reducing parameter count through things like knowledge distillation and pruning for future work. That tells you they know it’s not perfect right now.

Lu: Yeah, they are acknowledging that the current version is big, but they are clearly pointing toward making it more efficient without sacrificing the quality of the hierarchical representation.

Tom: So to wrap up this conclusion: HiVE-VQA successfully proves that structuring features hierarchically from a single powerful video model is an effective way to get better, more generalizable quality scores than current methods.

Jane: It suggests we can build better video quality assessment tools by focusing on how we organize the information inside the encoder rather than just swapping out whole models.

More episodes

← Home