HiVE-VQA: Hierarchical Interaction via Volumetric Embedding for Video Quality Assessment
summary
The gist
HiVE-VQA proposes HiVE-VQA, the first framework to apply a JEPA-based video encoder to this task It extracts token grids from multiple depths of a pre-trained V-JEPA 2.1 encoder and stacks them into…
In short
HiVE-VQA introduces a framework that uses a pre-trained V-JEPA 2.1 encoder to create hierarchical video features. It stacks these features into a 3D volume and fuses them using a compact 3D convolutional head. This method outperforms existing state-of-the-art models in video quality assessment by effectively capturing both local and global context from the video data.
Key concepts
- JEPA-based Video Encoder
- This is a self-supervised model pre-trained on videos that learns rich, general spatial and temporal features. HiVE-VQA repurposes this encoder to extract high-quality representations of video content, which are then used as the foundation for quality prediction.
- Hierarchical Feature Extraction
- The framework extracts video features from different depths of the pre-trained encoder and organizes them into a 3D volume. This structure allows the model to analyze visual information at multiple levels—from fine details to broader temporal patterns—for a more comprehensive quality assessment.
- Volumetric Quality Prediction
- A lightweight 3D convolutional head is used to fuse the hierarchical feature volume across its spatial and temporal dimensions. This fusion process combines local visual context with global video context to predict a single, accurate scalar score representing the video's quality.
Terminology used across episodes
This episode discusses
- HiVE-VQA: Hierarchical Interaction via Volumetric Embedding for Video Quality Assessment · Paper Radio
- V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- Distilling the Knowledge in a Neural Network
- V-JEPA 2.1: Unlocking Dense Features in Video Self-Supervised Learning
- Q-Align: Teaching LMMs for Visual Scoring via Discrete Text-Defined Levels
- Vision Transformer Pruning
The paper
HiVE-VQA: Hierarchical Interaction via Volumetric Embedding for Video Quality Assessment · Read on arXiv
University of Bristol
The proliferation of user-generated video content has made accurate and robust no-reference video quality assessment (NR-VQA) a highly important challenge. Existing research increasingly relies on architecturally complex multi-branch pipelines or computationally intensive fine-tuning of single-branch backbones. In this paper, we investigate whether a single frozen video foundation model can serve as an effective NR-VQA backbone, and propose HiVE-VQA, the first framework (to our knowledge) to apply a JEPA-based video encoder to this task. HiVE-VQA extracts token grids from multiple depths of a pre-trained V-JEPA 2.1 encoder and stacks them into a hierarchical spatio-temporal volume, which a compact 3D convolutional head fuses across its layer, spatial, and temporal axes, combining low-level distortion cues from shallow layers with the semantic and temporal context captured at greater depths. The encoder remains frozen throughout, so only the head is trained: 5.2M parameters (5.7% of the total), with no auxiliary multi-modal models and an inference cost that is invariant to video resolution. Trained solely on LSVQ and evaluated without any fine-tuning, HiVE-VQA outperforms 17 state-of-the-art (SOTA) methods in SROCC and PLCC on both intra-dataset (LSVQ-test, LSVQ-1080p) and cross-dataset (KoNViD-1k, LIVE-VQC) benchmarks. These results indicate that hierarchical, depth-wise fusion of features from a single frozen video foundation model provides a more effective and efficient basis for NR-VQA than task-specific multi-network fusion. Code and model checkpoints are available at https://github.com/RobHam99/HiVE-VQA.
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: We're looking at the paper: "HiVE-VQA: Hierarchical Interaction via Volumetric Embedding for Video Quality Assessment".
Jane: Here's the gist of the paper. HiVE-VQA proposes HiVE-VQA, the first framework to apply a JEPA-based video encoder to this task It extracts token grids from multiple depths of a pre-trained V-JEPA 2.1 encoder and stacks them into a hierarchical spatio-temporal volume, which a compact 3D convolutional head fuses across its layer, spatial, and temporal axes This approach demonstrates that hierarchical, depth-wise fusion of features from a single frozen video foundation model provides a more effective and efficient basis for NR-VQA than task-specific multinetwork fusion HiVE-VQA outperforms 17 stateof-the-art (SOTA) methods in SROCC and PLCC on both intra-dataset (LSVQ-test, LSVQ-1080p) and cross-dataset (KoNViD-1k, LIVE-VQC) benchmarks Framework Overview The proposed framework repurposes a self-supervised video encoder, pre-trained with the V-JEPA 2.1 objective
30: , as a hierarchical feature extractor for no-reference video quality assessment The framework consists of three stages: (i) Data sampling, which maps an input sequence X to a fixed-size fragment collage X' using the fragment sampling strategy introduced in
45: , (ii) Hierarchical feature extraction, where features are extracted from a uniformly spaced subset of L transformer blocks and folded into spatio-temporal grids to create a hierarchical feature volume V, and (iii) Volumetric quality prediction, where a lightweight 3D convolutional head fuses the hierarchical volume across its layer, spatial, and temporal axes to predict a single scalar quality score sˆ.
Tom: Training Signal Construction The model is trained using a hybrid loss function configured to optimize both the ranking monotonicity and linear correlation with human perceptual quality This loss is defined as L hybrid = ω rank L rank + ω PLCC L PLCC.
Jane: The ranking-based loss function, L rank, is designed to guide the model to learn the quality monotonicity of usergenerated content by defining s ij = sign(y i - y j) and prediction differences d ij = hat y i - hat y j.
Tom: The Differentiable Pearson Linear Correlation Coefficient (PLCC) losses are common in the quality assessment field, containing two terms: L MSE which minimizes the normalized mean squared error and L corr which aligns the prediction scale with the correlation coefficient.
Paper summary: Lu: The authors of HiVE-VQA are showing that this hierarchical feature extraction approach using the VJEPA two point one encoder can effectively represent low level quality information, and their proposed lightweight quality module captures both local and global context from a hierarchical feature volume <ref:2611.17146#pg1>.
Tom: It’s worth remembering that they achieved these results by repurposing an existing self supervised video encoder pre trained with the VJEPA two point one objective to serve as a hierarchical feature extractor for no reference video quality assessment <ref:2611.17146#pg1,no reference video quality assessment>.
Jane: So, in simple terms, this paper is about taking a powerful tool that already understands video very well and structuring its outputs in a layered way so you can assess how bad the quality is without needing perfect examples of good or bad videos.
Meng: It makes sense that they focused on fusing those features across layer, spatial, and temporal axes because they needed to capture both the immediate visual glitches and the broader context of what’s happening in the video.
Lalam: The main implication is that we don't always need entirely new architectures for every quality assessment problem; sometimes structuring existing powerful models hierarchically is enough to get accurate prediction, especially for user generated content.
Tom: They did show that this method achieves strong generalizability across both cross dataset and task specific evaluations, meaning it holds up well even when you move from one type of video quality test to another.
Jane: So, what we’re seeing here is a framework that uses existing powerful tools to build something specialized for no reference video quality assessment, which is a pretty significant direction for the field.
Conclusion: Tom: So we've been looking at HiVE-VQA, and now we’re getting to the conclusion of this paper titled "HiVE-VQA: Hierarchical Interaction via Volumetric Embedding for Video Quality Assessment."
Jane: Right, so what is the big picture here? Basically, the authors took an existing video encoder from a self supervised learning model and they layered its features up into a three dee volume to see if that helps us score video quality without needing any human labels.
Lu: The core idea they are pushing is that this hierarchical fusion of features across layer, spatial, and temporal axes provides a much better basis for no reference video quality assessment than just using a single feature vector from the final part of the encoder.
Meng: From my side, I’m thinking about deployment. If we can use this structure to get results like those reported on SROCC and PLCC, that means we might be able to run quality checks much faster than those other complex models you mentioned earlier.
Tom: Exactly, because they show that at 1080p resolution, this HiVE-VQA is about eleven times less computationally expensive than some of the state of the art methods we've seen.
Jane: That’s a huge number to drop on the listener—eleven times less computation means it could be practical for real time applications or high volume processing.
Lu: And they show that even when they fine tune this on user generated content, it still improves performance over the best multi branch model by about one point one percent in both ranking and correlation scores.
Tom: One point one percent is small, but when you compare it to the gains seen on pre training data, like the LSVQ test where they improved SROCC by a point seven four percent, that's meaningful.
Jane: It means this approach isn't just theoretical; it actually translates into measurable accuracy across different testing environments.
Meng: But what about the limitations? The authors mention focusing on reducing parameter count through things like knowledge distillation and pruning for future work. That tells you they know it’s not perfect right now.
Lu: Yeah, they are acknowledging that the current version is big, but they are clearly pointing toward making it more efficient without sacrificing the quality of the hierarchical representation.
Tom: So to wrap up this conclusion: HiVE-VQA successfully proves that structuring features hierarchically from a single powerful video model is an effective way to get better, more generalizable quality scores than current methods.
Jane: It suggests we can build better video quality assessment tools by focusing on how we organize the information inside the encoder rather than just swapping out whole models.
More episodes
- 2603.28113-The multiplicative Lipschitz bound as an inductive bias
- 2603.28595-Optimistic Actor-Critic with Parametric Policies for Linear Markov Decision Processes
- 2603.29765-Training-Free Dynamic Upcycling of Expert Language Models
- 2605.00778-Observable Performance Does Not Fully Reflect Adaptive System Organization: A Multi-Level Analysis of Gait Dynamics Under Occlusal Constraint
- 2605.06906-TraXion: Rethinking Pre-training Frameworks for Mobility and Beyond
- 2605.09165-Sparse Layers are Critical to Scaling Looped Language Models
- 2605.09439-Inverse Design for Conditional Distribution Matching
- 2605.10313-Signature Approach for Contextual Bandits with Nonlinear and Path-dependent Rewards: Regret Bound and Statistical Inference
- 2605.10878-Minimum Neural Weight Norm = Kolmogorov Complexity
- 2605.11617-MIST: Reliable Streaming Decision Trees for Online Class-Incremental Learning via McDiarmid Bound