HiVE-VQA: Hierarchical Interaction via Volumetric Embedding for Video Quality Assessment

arXiv:2611.17146 · cs.CV · Submitted 2026-11-12 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "HiVE-VQA: Hierarchical Interaction via Volumetric Embedding for Video Quality Assessment".

Jane: Here's the gist of the paper. HiVE-VQA proposes HiVE-VQA, the first framework to apply a JEPA-based video encoder to this task It extracts token grids from multiple depths of a pre-trained V-JEPA 2.1 encoder and stacks them into a hierarchical spatio-temporal volume, which a compact 3D convolutional head fuses across its layer, spatial, and temporal axes This approach demonstrates that hierarchical, depth-wise fusion of features from a single frozen video foundation model provides a more effective and efficient basis for NR-VQA than task-specific multinetwork fusion HiVE-VQA outperforms 17 stateof-the-art (SOTA) methods in SROCC and PLCC on both intra-dataset (LSVQ-test, LSVQ-1080p) and cross-dataset (KoNViD-1k, LIVE-VQC) benchmarks Framework Overview The proposed framework repurposes a self-supervised video encoder, pre-trained with the V-JEPA 2.1 objective

30: , as a hierarchical feature extractor for no-reference video quality assessment The framework consists of three stages: (i) Data sampling, which maps an input sequence X to a fixed-size fragment collage X' using the fragment sampling strategy introduced in

45: , (ii) Hierarchical feature extraction, where features are extracted from a uniformly spaced subset of L transformer blocks and folded into spatio-temporal grids to create a hierarchical feature volume V, and (iii) Volumetric quality prediction, where a lightweight 3D convolutional head fuses the hierarchical volume across its layer, spatial, and temporal axes to predict a single scalar quality score sˆ.

Tom: Training Signal Construction The model is trained using a hybrid loss function configured to optimize both the ranking monotonicity and linear correlation with human perceptual quality This loss is defined as L hybrid = ω rank L rank + ω PLCC L PLCC.

Jane: The ranking-based loss function, L rank, is designed to guide the model to learn the quality monotonicity of usergenerated content by defining s ij = sign(y i - y j) and prediction differences d ij = hat y i - hat y j.

Tom: The Differentiable Pearson Linear Correlation Coefficient (PLCC) losses are common in the quality assessment field, containing two terms: L MSE which minimizes the normalized mean squared error and L corr which aligns the prediction scale with the correlation coefficient.

Paper summary: Lu: The authors of HiVE-VQA are showing that this hierarchical feature extraction approach using the VJEPA two point one encoder can effectively represent low level quality information, and their proposed lightweight quality module captures both local and global context from a hierarchical feature volume <ref:2611.17146#pg1>.

Tom: It’s worth remembering that they achieved these results by repurposing an existing self supervised video encoder pre trained with the VJEPA two point one objective to serve as a hierarchical feature extractor for no reference video quality assessment <ref:2611.17146#pg1,no reference video quality assessment>.

Jane: So, in simple terms, this paper is about taking a powerful tool that already understands video very well and structuring its outputs in a layered way so you can assess how bad the quality is without needing perfect examples of good or bad videos.

Meng: It makes sense that they focused on fusing those features across layer, spatial, and temporal axes because they needed to capture both the immediate visual glitches and the broader context of what’s happening in the video.

Lalam: The main implication is that we don't always need entirely new architectures for every quality assessment problem; sometimes structuring existing powerful models hierarchically is enough to get accurate prediction, especially for user generated content.

Tom: They did show that this method achieves strong generalizability across both cross dataset and task specific evaluations, meaning it holds up well even when you move from one type of video quality test to another.

Jane: So, what we’re seeing here is a framework that uses existing powerful tools to build something specialized for no reference video quality assessment, which is a pretty significant direction for the field.

Conclusion: Tom: So we've been looking at HiVE-VQA, and now we’re getting to the conclusion of this paper titled "HiVE-VQA: Hierarchical Interaction via Volumetric Embedding for Video Quality Assessment."

Jane: Right, so what is the big picture here? Basically, the authors took an existing video encoder from a self supervised learning model and they layered its features up into a three dee volume to see if that helps us score video quality without needing any human labels.

Lu: The core idea they are pushing is that this hierarchical fusion of features across layer, spatial, and temporal axes provides a much better basis for no reference video quality assessment than just using a single feature vector from the final part of the encoder.

Meng: From my side, I’m thinking about deployment. If we can use this structure to get results like those reported on SROCC and PLCC, that means we might be able to run quality checks much faster than those other complex models you mentioned earlier.

Tom: Exactly, because they show that at 1080p resolution, this HiVE-VQA is about eleven times less computationally expensive than some of the state of the art methods we've seen.

Jane: That’s a huge number to drop on the listener—eleven times less computation means it could be practical for real time applications or high volume processing.

Lu: And they show that even when they fine tune this on user generated content, it still improves performance over the best multi branch model by about one point one percent in both ranking and correlation scores.

Tom: One point one percent is small, but when you compare it to the gains seen on pre training data, like the LSVQ test where they improved SROCC by a point seven four percent, that's meaningful.

Jane: It means this approach isn't just theoretical; it actually translates into measurable accuracy across different testing environments.

Meng: But what about the limitations? The authors mention focusing on reducing parameter count through things like knowledge distillation and pruning for future work. That tells you they know it’s not perfect right now.

Lu: Yeah, they are acknowledging that the current version is big, but they are clearly pointing toward making it more efficient without sacrificing the quality of the hierarchical representation.

Tom: So to wrap up this conclusion: HiVE-VQA successfully proves that structuring features hierarchically from a single powerful video model is an effective way to get better, more generalizable quality scores than current methods.

Jane: It suggests we can build better video quality assessment tools by focusing on how we organize the information inside the encoder rather than just swapping out whole models.

University of Bristol

cs.CV

Submitted: 2026-11-12

Updated: 2026-11-12

Comments: Accepted to WACV 2027, 11 pages, 3 figures, 6 tables

Code: https://github.com/RobHam99/HiVE-VQA

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 90/100

The gist: HiVE-VQA proposes HiVE-VQA, the first framework to apply a JEPA-based video encoder to this task It extracts token grids from multiple depths of a pre-trained V-JEPA 2.1 encoder and stacks them into…

Key concepts

JEPA-based Video Encoder
This is a self-supervised model pre-trained on videos that learns rich, general spatial and temporal features. HiVE-VQA repurposes this encoder to extract high-quality representations of video content, which are then used as the foundation for quality prediction.
Hierarchical Feature Extraction
The framework extracts video features from different depths of the pre-trained encoder and organizes them into a 3D volume. This structure allows the model to analyze visual information at multiple levels—from fine details to broader temporal patterns—for a more comprehensive quality assessment.
Volumetric Quality Prediction
A lightweight 3D convolutional head is used to fuse the hierarchical feature volume across its spatial and temporal dimensions. This fusion process combines local visual context with global video context to predict a single, accurate scalar score representing the video's quality.

Terminology

Summary

HiVE-VQA proposes HiVE-VQA, the first framework to apply a JEPA-based video encoder to this task It extracts token grids from multiple depths of a pre-trained V-JEPA 2. p. 0

Where the paper says it

In this paper, we investigate whether a single frozen video foundation model can serve as an effective NR-VQA backbone, and propose HiVE-VQA, the first framework (to our knowledge) to apply a JEPA-based video encoder to this task. HiVE-VQA extracts token grids from multiple depths of a pre-trained V-JEPA 2.

1 encoder and stacks them into a hierarchical spatio-temporal volume, which a compact 3D convolutional head fuses across its layer, spatial, and temporal axes This approach demonstrates that hierarchical, depth-wise fusion of features from a single frozen video foundation model provides a more effective and efficient basis for NR-VQA than task-specific multinetwork fusion HiVE-VQA outperforms 17 stateof-the-art (SOTA) methods in SROCC and PLCC on both intra-dataset (LSVQ-test, LSVQ-1080p) and cross-dataset (KoNViD-1k, LIVE-VQC) benchmarks

Framework Overview

The proposed framework repurposes a self-supervised video encoder, pre-trained with the V-JEPA 2.1 objective [30], as a hierarchical feature extractor for no-reference video quality assessment The framework consists of three stages: (i) Data sampling, which maps an input sequence X to a fixed-size fragment collage X' using the fragment sampling strategy introduced in [45], (ii) Hierarchical feature extraction, where features are extracted from a uniformly spaced subset of L transformer blocks and folded into spatio-temporal grids to create a hierarchical feature volume V, and (iii) Volumetric quality prediction, where a lightweight 3D convolutional head fuses the hierarchical volume across its layer, spatial, and temporal axes to predict a single scalar quality score sˆ.

Training Signal Construction

The model is trained using a hybrid loss function configured to optimize both the ranking monotonicity and linear correlation with human perceptual quality This loss is defined as L hybrid = ω rank L rank + ω PLCC L PLCC. The ranking-based loss function, L rank, is designed to guide the model to learn the quality monotonicity of usergenerated content by defining s ij = sign(y i - y j) and prediction differences d ij = hat y i - hat y j. p. 3

Where the paper says it

\mathcal {L}_{\text {hybrid}}=\omega _{\text {rank}}\mathcal {L}_{\text {rank}}+\omega _{\text {PLCC}}\mathcal {L}_{\text {PLCC}}, \label {eq:hybrid_loss} (1) where the ranking-based loss function Lrank is designed to guide the model to learn the quality monotonicity of usergenerated content. For all valid pairs (i, j) such that yi ̸= yj , we define: s_{ij}=\text {sign}(y_i-y_j), (2) and prediction differences: d_{ij}=\hat {y}_i-\hat {y}_j.

The Differentiable Pearson Linear Correlation Coefficient (PLCC) losses are common in the quality assessment field, containing two terms: L MSE which minimizes the normalized mean squared error and L corr which aligns the prediction scale with the correlation coefficient. p. 4
Where the paper says it

The first term minimizes the normalized mean squared error: \mathcal {L}_{\mathrm {MSE}} = \frac {1}{4N} \sum _{i=1}^N (\tilde y_i - \tilde t_i)^2. (8) The second term aligns the prediction scale with the correlation coefficient: \mathcal {L}_{\mathrm {corr}} = \frac {1}{4N} \sum _{i=1}^N (\rho \tilde y_i - \tilde t_i)^2.

Performance Gains Analysis

The results show that HiVE-VQA achieves strong generalizability and accurate quality prediction across both cross-dataset and task-specific evaluations In pre-training results on LSVQ, the model outperforms all of the classical VQA metrics included in the benchmark, improving mean SROCC and PLCC by 1.74% and 1.60%, respectively. p. 4

Where the paper says it

HiVEVQA is pre-trained on LSVQtrain, which is a subset of the main LSVQ dataset comprising 28,056 video clips. After pre-training, intra-dataset evaluation is performed on the two LSVQ test datasets, which contain 7,400 and 3,600 videos, respectively.

When fine-tuned on UGC datasets, HiVE-VQA achieves the best average performance among all models, improving SROCC and PLCC by 0. p. 5
Where the paper says it

We repeat the fine-tuning process 10 times across a 70/10/20% dataset split (train/val/test), and report the median performance across the 10 runs in Table 2. It is noted that HiVE-VQA once again achieves the best average performance among all models, improving SROCC and PLCC by 0.

11% and 1.10%, respectively, over the best multi-branch model. p. 5
Where the paper says it

It is noted that HiVE-VQA once again achieves the best average performance among all models, improving SROCC and PLCC by 0. 10%, respectively, over the best multi-branch model.

Encoder Representation Analysis

Analysis of Embedding Sensitivity shows that each type of distortion produces features with distinct representations, indicating that the V-JEPA 2.1 is indeed sensitive to various distortions Layer selection analysis suggests that hierarchical fusion offers an improvement (+0. p. 6

Where the paper says it

1 is indeed sensitive to various distortions. Layer selection analysis.

012 SROCC) over the standard method of extracting features only from the final layer. p. 7
Where the paper says it

1 ViT-B) 0. 890 the standard method of extracting features only from the final layer.

Ablation studies confirm that replacing the V-JEPA 2. p. 7
Where the paper says it

Ablation Studies Volumetric Embedding Network Ablation. We verify the effectiveness of the volumetric network by replacing it with a ridge regressor trained on the same multi-scale features and LSVQ training dataset (Table 6).

1 trained ViT-B encoder with one image pre-trained ViT-B [44] and VideoMAE ViT-B [36] shows that latent JEPA features greatly enhance performance with mean SROCC gains of +0. p. 7
Where the paper says it

We train each replacement encoder under identical setups with the same HiVE-VQA framework. This ablation study shows that latent JEPA features greatly enhance performance with mean SROCC gains of +0.

080 and +0. p. 1
Where the paper says it

1 backbone provides a substantially stronger fixed representation than a comparably sized image-pretrained ViT (+0. 080 average SROCC, Table 6), requiring no encoder fine-tuning and only 5.

105, respectively. p. 7
Where the paper says it

080 and +0. 105, respectively.

Efficiency Considerations

HiVE-VQA is evaluated by comparing both variants with other SOTA NR-VQA models, showing that at 1080p HiVE-VQA is roughly 11×, 15×, and 29× less computationally expensive than VSFA, PVQ, and BVQA respectively. The smaller variant, HiVE-VQA-M, is on the same order of FLOPs as the most lightweight model, FasterVQA. p. 6

Where the paper says it

2M trainable parameters. Our smaller variant, HiVE-VQA-M, is on the same order of FLOPs as the most lightweight model, FasterVQA, while yielding average SROCC and PLCC gains of +0.

This analysis indicates HiVE-VQA is a promising alternative for realistic implementations of UGC NRVQA. p. 6
Where the paper says it

025 respectively. This analysis indicates HiVE-VQA is a promising alternative for realistic implementations of UGC NRVQA.

Conclusion

In this paper, the authors are to the best of their knowledge, the first to employ a JEPA-based encoder for NR-VQA, demonstrating its enhanced capability to represent low-level quality information They propose a lightweight quality module that effectively captures both local and global context from a hierarchical feature volume Extensive experiments demonstrate that the HiVE-VQA model achieves strong generalizability and accurate quality prediction across both cross-dataset and task-specific evaluations, with predictions that correlate well with human subjective judgments. Future work will focus on reducing the total parameter count through techniques such as knowledge distillation [9] and model pruning [14, 53]. p. 7

Where the paper says it

Extensive experiments demonstrate that the HiVE-VQA model achieves strong generalizability and accurate quality prediction across both cross-dataset and task-specific evaluations, with predictions that correlate well with human subjective judgments. Future work will focus on reducing the total parameter count through techniques such as knowledge distillation [9] and model pruning [14, 53].

Acknowledgments

The authors appreciate the funding from the EPSRC Doctoral Landscape Award, the University of Bristol, and the UKRI MyWorld Strength in Places Program (SIPF00006/1) They would like to thank Benoit Vallade and Alex Mackin from Amazon Prime Video, UK, for their valuable advice and support throughout this work. The authors also acknowledge the use of resources provided by the Isambard-AI National AI Research Resource (AIRR). p. 8

Where the paper says it

We would like to thank Benoit Vallade and Alex Mackin from Amazon Prime Video, UK, for their valuable advice and support throughout this work. We also acknowledge the use of resources provided by the Isambard-AI National AI Research Resource (AIRR).

They are funded by the UK Government’s Department for Science, Innovation and Technology (DSIT) via UK Research and Innovation; and the Science and Technology Facilities Council [ST/AIRR/I-A-I/1023]. p. 8
Where the paper says it

We also acknowledge the use of resources provided by the Isambard-AI National AI Research Resource (AIRR). Isambard-AI is operated by the University of Bristol and is funded by the UK Government’s Department for Science, Innovation and Technology (DSIT) via UK Research and Innovation; and the Science and Technology Facilities Council [ST/AIRR/I-A-I/1023].

The references listed are [1] through [53]. p. 8
Where the paper says it

Isambard-AI is operated by the University of Bristol and is funded by the UK Government’s Department for Science, Innovation and Technology (DSIT) via UK Research and Innovation; and the Science and Technology Facilities Council [ST/AIRR/I-A-I/1023]. References [1] Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, Yann LeCun, and Nicolas Ballas.

References

[540p 720p 89M 89M 2406G 93M Time GFLOPs Time GFLOPs Time GFLOPs Time GFLOPs] [5.2 M parameters (5. p. 0

Where the paper says it

The encoder remains frozen throughout, so only the head is trained: 5. 2M parameters (5.

7% of the total), with no auxiliary multimodal models and an inference cost that is invariant to video resolution Code and model checkpoints are available at https://github. p. 0
Where the paper says it

2M parameters (5. 7% of the total), with no auxiliary multimodal models and an inference cost that is invariant to video resolution.

com/RobHam99/HiVE-VQA [30] Lorenzo Mur-Labadia, Matthew Muckley, Amir Bar, Mahmoud Assran, Koustuv Sinha, Michael Rabbat, Yann LeCun, Nicolas Ballas, and Adrien Bardes. p. 9
Where the paper says it

IEEE Transactions on Image Processing, 25(1):289–300, 2016. 1, 2 [30] Lorenzo Mur-Labadia, Matthew Muckley, Amir Bar, Mahmoud Assran, Koustuv Sinha, Michael Rabbat, Yann LeCun, Nicolas Ballas, and Adrien Bardes.

V-jepa 2.1: Unlocking dense features in video self-supervised learning [3] Baoliang Chen, Lingyu Zhu, Guo Li, Fangbo Lu, Hongfei Fan, and Shiqi Wang. p. 8
Where the paper says it

2 [3] Baoliang Chen, Lingyu Zhu, Guo Li, Fangbo Lu, Hongfei Fan, and Shiqi Wang. Learning generalized spatial-temporal deep feature representation for no-reference video quality assessment.

p. 8
Where the paper says it

2 [3] Baoliang Chen, Lingyu Zhu, Guo Li, Fangbo Lu, Hongfei Fan, and Shiqi Wang. Learning generalized spatial-temporal deep feature representation for no-reference video quality assessment.

p. 8
Where the paper says it

2 [3] Baoliang Chen, Lingyu Zhu, Guo Li, Fangbo Lu, Hongfei Fan, and Shiqi Wang. Learning generalized spatial-temporal deep feature representation for no-reference video quality assessment.

p. 8
Where the paper says it

2 [3] Baoliang Chen, Lingyu Zhu, Guo Li, Fangbo Lu, Hongfei Fan, and Shiqi Wang. Learning generalized spatial-temporal deep feature representation for no-reference video quality assessment.

p. 8
Where the paper says it

2 [3] Baoliang Chen, Lingyu Zhu, Guo Li, Fangbo Lu, Hongfei Fan, and Shiqi Wang. Learning generalized spatial-temporal deep feature representation for no-reference video quality assessment.

p. 8
Where the paper says it

2 [3] Baoliang Chen, Lingyu Zhu, Guo Li, Fangbo Lu, Hongfei Fan, and Shiqi Wang. Learning generalized spatial-temporal deep feature representation for no-reference video quality assessment.

p. 8
Where the paper says it

2 [3] Baoliang Chen, Lingyu Zhu, Guo Li, Fangbo Lu, Hongfei Fan, and Shiqi Wang. Learning generalized spatial-temporal deep feature representation for no-reference video quality assessment.

p. 8
Where the paper says it

2 [3] Baoliang Chen, Lingyu Zhu, Guo Li, Fangbo Lu, Hongfei Fan, and Shiqi Wang. Learning generalized spatial-temporal deep feature representation for no-reference video quality assessment.

Learning generalized spatial-temporal deep feature representation for no-reference video quality assessment [5] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. p. 8
Where the paper says it

html, 2020. 1 [5] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.

p. 8
Where the paper says it

html, 2020. 1 [5] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.

p. 8
Where the paper says it

html, 2020. 1 [5] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.

p. 8
Where the paper says it

html, 2020. 1 [5] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.

p. 8
Where the paper says it

html, 2020. 1 [5] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.

p. 8
Where the paper says it

html, 2020. 1 [5] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.

p. 8
Where the paper says it

html, 2020. 1 [5] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.

p. 8
Where the paper says it

html, 2020. 1 [5] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.

An image is worth 16x16 words: Transformers for image recognition at scale [39] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. p. 9
Where the paper says it

IEEE Open Journal of Signal Processing, 2:425–440, 2021. 1, 2, 7 [39] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.

p. 9
Where the paper says it

IEEE Open Journal of Signal Processing, 2:425–440, 2021. 1, 2, 7 [39] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.

p. 9
Where the paper says it

IEEE Open Journal of Signal Processing, 2:425–440, 2021. 1, 2, 7 [39] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.

p. 9
Where the paper says it

IEEE Open Journal of Signal Processing, 2:425–440, 2021. 1, 2, 7 [39] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.

p. 9
Where the paper says it

IEEE Open Journal of Signal Processing, 2:425–440, 2021. 1, 2, 7 [39] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.

p. 9
Where the paper says it

IEEE Open Journal of Signal Processing, 2:425–440, 2021. 1, 2, 7 [39] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.

p. 9
Where the paper says it

IEEE Open Journal of Signal Processing, 2:425–440, 2021. 1, 2, 7 [39] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.

p. 9
Where the paper says it

IEEE Open Journal of Signal Processing, 2:425–440, 2021. 1, 2, 7 [39] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.

Attention is all you need [40] Video Quality Experts Group. p. 9
Where the paper says it

1, 2, 7 [39] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need.

p. 9
Where the paper says it

1, 2, 7 [39] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need.

p. 9
Where the paper says it

1, 2, 7 [39] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need.

p. 9
Where the paper says it

1, 2, 7 [39] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need.

p. 9
Where the paper says it

1, 2, 7 [39] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need.

p. 9
Where the paper says it

1, 2, 7 [39] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need.

p. 9
Where the paper says it

1, 2, 7 [39] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need.

p. 9
Where the paper says it

1, 2, 7 [39] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need.

Final report from the video quality experts group on the validation of objective quality metrics for video quality assessment [53] Mingjian Zhu, Yehui Tang, and Kai Han. p. 9
Where the paper says it

3 [40] Video Quality Experts Group. Final report from the video quality experts group on the validation of objective quality metrics for video quality assessment.

p. 9
Where the paper says it

3 [40] Video Quality Experts Group. Final report from the video quality experts group on the validation of objective quality metrics for video quality assessment.

p. 9
Where the paper says it

3 [40] Video Quality Experts Group. Final report from the video quality experts group on the validation of objective quality metrics for video quality assessment.

p. 9
Where the paper says it

3 [40] Video Quality Experts Group. Final report from the video quality experts group on the validation of objective quality metrics for video quality assessment.

p. 9
Where the paper says it

3 [40] Video Quality Experts Group. Final report from the video quality experts group on the validation of objective quality metrics for video quality assessment.

p. 9
Where the paper says it

3 [40] Video Quality Experts Group. Final report from the video quality experts group on the validation of objective quality metrics for video quality assessment.

p. 9
Where the paper says it

3 [40] Video Quality Experts Group. Final report from the video quality experts group on the validation of objective quality metrics for video quality assessment.

Vision transformer pruning [30] Lorenzo Mur-Labadia, Matthew Muckley, Amir Bar, Mahmoud Assran, Koustuv Sinha, Michael Rabbat, Yann LeCun, Nicolas Ballas, and Adrien Bardes. p. 9
Where the paper says it

IEEE Transactions on Image Processing, 25(1):289–300, 2016. 1, 2 [30] Lorenzo Mur-Labadia, Matthew Muckley, Amir Bar, Mahmoud Assran, Koustuv Sinha, Michael Rabbat, Yann LeCun, Nicolas Ballas, and Adrien Bardes.

p. 9
Where the paper says it

IEEE Transactions on Image Processing, 25(1):289–300, 2016. 1, 2 [30] Lorenzo Mur-Labadia, Matthew Muckley, Amir Bar, Mahmoud Assran, Koustuv Sinha, Michael Rabbat, Yann LeCun, Nicolas Ballas, and Adrien Bardes.

p. 9
Where the paper says it

IEEE Transactions on Image Processing, 25(1):289–300, 2016. 1, 2 [30] Lorenzo Mur-Labadia, Matthew Muckley, Amir Bar, Mahmoud Assran, Koustuv Sinha, Michael Rabbat, Yann LeCun, Nicolas Ballas, and Adrien Bardes.

p. 9
Where the paper says it

IEEE Transactions on Image Processing, 25(1):289–300, 2016. 1, 2 [30] Lorenzo Mur-Labadia, Matthew Muckley, Amir Bar, Mahmoud Assran, Koustuv Sinha, Michael Rabbat, Yann LeCun, Nicolas Ballas, and Adrien Bardes.

p. 9
Where the paper says it

IEEE Transactions on Image Processing, 25(1):289–300, 2016. 1, 2 [30] Lorenzo Mur-Labadia, Matthew Muckley, Amir Bar, Mahmoud Assran, Koustuv Sinha, Michael Rabbat, Yann LeCun, Nicolas Ballas, and Adrien Bardes.

p. 9
Where the paper says it

IEEE Transactions on Image Processing, 25(1):289–300, 2016. 1, 2 [30] Lorenzo Mur-Labadia, Matthew Muckley, Amir Bar, Mahmoud Assran, Koustuv Sinha, Michael Rabbat, Yann LeCun, Nicolas Ballas, and Adrien Bardes.

p. 9
Where the paper says it

IEEE Transactions on Image Processing, 25(1):289–300, 2016. 1, 2 [30] Lorenzo Mur-Labadia, Matthew Muckley, Amir Bar, Mahmoud Assran, Koustuv Sinha, Michael Rabbat, Yann LeCun, Nicolas Ballas, and Adrien Bardes.

V-jepa 2.1: Unlocking dense features in video self-supervised learning [3] Baoliang Chen, Lingyu Zhu, Guo Li, Fangbo Lu, Hongfei Fan, and Shiqi Wang. Learning generalized spatial-temporal deep feature representation for no-reference video quality assessment [5] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale [39] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need [40] Video Quality Experts Group. Final report from the video quality experts group on the validation of objective quality metrics for video quality assessment [53] Mingjian Zhu, Yehui Tang, and Kai Han. Vision transformer pruning [30] Lorenzo Mur-Labadia, Matthew Muckley, Amir Bar, Mahmoud Assran, Koustuv Sinha, Michael Rabbat, Yann LeCun, Nicolas Ballas, and Adrien Bardes. V-jepa 2.1: Unlocking dense features in video self-supervised learning [3] Baoliang Chen, Lingyu Zhu, Guo Li, Fangbo Lu, Hongfei Fan, and Shiqi Wang. Learning generalized spatial-temporal deep feature representation for no-reference video quality assessment [5] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale [39] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need [40] Video Quality Experts Group. Final report from the video quality experts group on the validation of objective quality metrics for video quality assessment [53] Mingjian Zhu, Yehui Tang, and Kai Han. Vision transformer pruning [30] Lorenzo Mur-Labadia, Matthew Muckley, Amir Bar, Mahmoud Assran, Koustuv Sinha, Michael Rabbat, Yann LeCun, Nicolas Ballas, and Adrien Bardes. V-jepa 2.1: Unlocking dense features in video self-supervised learning [3] Baoliang Chen, Lingyu Zhu, Guo Li, Fangbo Lu, Hongfei Fan, and Shiqi Wang. Learning generalized spatial-temporal deep feature representation for no-reference video quality assessment [5] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale [39] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need [40] Video Quality Experts Group. Final report from the video quality experts group on the validation of objective quality metrics for video quality assessment [53] Mingjian Zhu, Yehui Tang, and Kai Han. Vision transformer pruning [30] Lorenzo Mur-Labadia, Matthew Muckley, Amir Bar, Mahmoud Assran, Koustuv Sinha, Michael Rabbat, Yann LeCun, Nicolas Ballas, and Adrien Bardes. V-jepa 2.1: Unlocking dense features in video self-supervised learning [3] Baoliang Chen, Lingyu Zhu, Guo Li, Fangbo Lu, Hongfei Fan, and Shiqi Wang. Learning generalized spatial-temporal deep feature representation for no-reference video quality assessment [5] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale [39] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need [40] Video Quality Experts Group. Final report from the video quality experts group on the validation of objective quality metrics for video quality assessment [53] Mingjian Zhu, Yehui Tang, and Kai Han. Vision transformer pruning [30] Lorenzo Mur-Labadia, Matthew Muckley, Amir Bar, Mahmoud Assran, Koustuv Sinha, Michael Rabbat, Yann LeCun, Nicolas Ballas, and Adrien Bardes. V-jepa 2.1: Unlocking dense features in video self-supervised learning [3] Baoliang Chen, Lingyu Zhu, Guo Li, Fangbo Lu, Hongfei Fan, and Shiqi Wang. Learning generalized spatial-temporal deep feature representation for no-reference video quality assessment [5] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale [39] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need [40] Video Quality Experts Group. Final report from the video quality experts group on the validation of objective quality metrics for video quality assessment [53] Mingjian Zhu, Yehui Tang, and Kai Han. Vision transformer pruning [30] Lorenzo Mur-Labadia, Matthew Muckley, Amir Bar, Mahmoud Assran, Koustuv Sinha, Michael Rabbat, Yann LeCun, Nicolas Ballas, and Adrien Bardes. V-jepa 2.1: Unlocking dense features in video self-supervised learning [3] Baoliang Chen, Lingyu Zhu, Guo Li, Fangbo Lu, Hongfei Fan, and Shiqi Wang. Learning generalized spatial-temporal deep feature representation for no-reference video quality assessment [5] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale [39] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need [40] Video Quality Experts Group. Final report from the video quality experts group on the validation of objective quality metrics for video quality assessment [53] Mingjian Zhu, Yehui Tang, and Kai Han. Vision transformer pruning [30] Lorenzo Mur-Labadia, Matthew Muckley, Amir Bar, Mahmoud Assran, Koustuv Sinha, Michael Rabbat, Yann LeCun, Nicolas Ballas, and Adrien Bardes. V-jepa 2.1: Unlocking dense features in video self-supervised learning [3] Baoliang Chen, Lingyu Zhu, Guo Li, Fangbo Lu, Hongfei Fan, and Shiqi Wang. Learning generalized spatial-temporal deep feature representation for no-reference video quality assessment [5] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale [39] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need [40] Video Quality Experts Group.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Hierarchical Feature Extraction for NR-VQA: The system can extract token grids from multiple depths of a pre-trained V-JEPA 2.1 encoder and stack them into a hierarchical spatio-temporal volume to combine low-level distortion cues with semantic context.

  2. Lightweight Prediction Module: A compact 3D convolutional head fuses the hierarchical volume across its layer, spatial, and temporal axes to predict a single quality score s hat s that is invariant to video resolution.

  3. Training Strategy for Efficiency: The system can be trained solely on the head, which has only 5.2 million parameters (5.7% of the total), resulting in an inference cost that is invariant to video resolution and no auxiliary multimodal models are required.

  4. Hybrid Loss Function Optimization: The model can be trained using a hybrid loss function combining ranking monotonicity loss Lrank and differentiable Pearson Linear Correlation Coefficient PLCC loss to optimize both quality ranking and linear alignment with human perceptual ratings.

  5. Generalizability Across Data Types: The framework demonstrates strong generalizability, achieving top performance across four UGC benchmarks (LSVQ1080p, LSVQtest, KoNViD-1k, LIVE-VQC) and showing competitive fine-tuning performance on multiple datasets.

Abstract

The proliferation of user-generated video content has made accurate and robust no-reference video quality assessment (NR-VQA) a highly important challenge. Existing research increasingly relies on architecturally complex multi-branch pipelines or computationally intensive fine-tuning of single-branch backbones. In this paper, we investigate whether a single frozen video foundation model can serve as an effective NR-VQA backbone, and propose HiVE-VQA, the first framework (to our knowledge) to apply a JEPA-based video encoder to this task. HiVE-VQA extracts token grids from multiple depths of a pre-trained V-JEPA 2.1 encoder and stacks them into a hierarchical spatio-temporal volume, which a compact 3D convolutional head fuses across its layer, spatial, and temporal axes, combining low-level distortion cues from shallow layers with the semantic and temporal context captured at greater depths. The encoder remains frozen throughout, so only the head is trained: 5.2M parameters (5.7% of the total), with no auxiliary multi-modal models and an inference cost that is invariant to video resolution. Trained solely on LSVQ and evaluated without any fine-tuning, HiVE-VQA outperforms 17 state-of-the-art (SOTA) methods in SROCC and PLCC on both intra-dataset (LSVQ-test, LSVQ-1080p) and cross-dataset (KoNViD-1k, LIVE-VQC) benchmarks. These results indicate that hierarchical, depth-wise fusion of features from a single frozen video foundation model provides a more effective and efficient basis for NR-VQA than task-specific multi-network fusion. Code and model checkpoints are available at https://github.com/RobHam99/HiVE-VQA.

Sources

Related papers