Deep Learning-based Filtering for Video Coding: A Survey on Architectures, Algorithms, and Complexity Analysis

arXiv:2607.16319 · cs.CV · Submitted 2026-07-15 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "Deep Learning-based Filtering for Video Coding".

Jane: Here's the gist of the paper. Deep Learning-based Filtering (DLF) has emerged as a promising solution to mitigate compression artifacts inherent in standards like High Efficiency Video Coding (HEVC/H.265) and Versatile Video Coding (VVC/H.266), but its deployment is severely constrained by computational complexity, memory bandwidth, and power consumption Systematic Classification of DLF Methods The paper proposes a systematic three-dimensional taxonomy classifying methods into (1) Integration Scheme within the Video Coding, (2) Coding Information Utilization, and (3) Network Design Strategy This taxonomy organizes the DLF design space along three axes that each govern a distinct CE deployment constraint Dimension 1 determines pipeline latency and bitstream-syntax dependency, Dimension 2 determines parsing dependency and memory bandwidth, and Dimension 3 determines per-pixel MACs and on-chip storage Analysis of Integration Schemes The integration scheme dimension is divided into Post-Processing Filtering (PPF), In-Loop Filtering (ILF), and Fractional-pixel Interpolation Filtering (FIF) PPF operates only at the decoder side as a post-processor, focusing on visual enhancement without affecting the coding loop ILF is embedded within the coding loop, where filtered frames serve as reference frames FIF replaces the interpolation filter in the ME/MC module and is invoked frequently at the block level, making its per-pixel complexity constraints significantly stricter than frame-level filters Utilization of Coding Information The second dimension explores how various information sources are utilized to enhance filtering performance Key differentiators include spatial context, temporal context, and utilization of coding priors Spatial context focuses on utilizing spatial information within a single frame to restore textures lost due to quantization Temporal context involves multi-frame fusion strategies that exploit temporal redundancy inherent in video data Utilization of coding priors includes using partition information, prediction signals, and QP/Mode information to guide the network toward artifact-prone regions Network Design Strategies for Hardware Feasibility The third dimension discusses the network design strategies and structural innovations employed in StateOf-The-Art (SOTA) DLF models Key building blocks include the Residual Block, Dense Block, Recursive Block, and Attention Block Structural priors such as Residual Learning are standard to model the sparse difference between distorted and original signals Attention mechanisms like Channel Attention explicitly model inter-dependencies between channels, emphasizing feature maps that contain relevant artifact information Complexity-oriented designs include Dynamic & Early Exit strategies where easy samples exit the network at early layers to reduce average computation Performance and Complexity Trade-offs The paper presents an extended trade-off analysis comparing HEVC Test Model (HM), VVC Test Model (VTM), and Enhanced Compression Model (ECM) across various coding configurations The evolution shows a paradigm shift towards ultra-lightweight architectures (< 50k MACs/pixel) to satisfy strict hardware constraints Ablation studies show that removing the explicit partitioning map or boundary strength input from the network resulted in negligible performance loss while reducing computational complexity The latest research trends demonstrate a paradigm shift towards ultra-lightweight architectures (< 50k MACs/pixel) to satisfy the strict hardware constraints of emerging standards like JVET NNVC Standardization and Future Challenges For DL technologies to be adopted in future standards, they must satisfy stringent engineering requirements such as integer-only fixed-point implementation via frameworks like SADL Strict caps on MACs/pixel and the elimination of line-buffering are critical for minimizing on-chip memory usage Open challenges include real-time inference latency, error propagation in ILF, domain generalization, and NPU-friendly model design Future work should explore meta-learning or few-shot adaptation techniques to allow the filter to adapt to new video domains with minimal data and updates Temporal consistency and error propagation control are key concerns for PPF and ILF, requiring mechanisms like bi-directional prediction or Kalman filtering to suppress flickering The primary challenge is the prohibitive computational cost and memory bandwidth of deep models DLF is poised to become an indispensable component of next-generation video coding systems with the advancement of DL-specific accelerators (NPUs) Intelligent video coding satisfying high performance, low power, and low latency will soon become a reality in IoT and CE The key findings are that DLFs have demonstrated remarkable capabilities in mitigating complex non-linear artifacts that traditional handcrafted filters fail to address Despite performance gains, computational complexity and memory bandwidth remain the primary bottlenecks for commercial deployment Building upon these foundational heavy models, the natural evolution of current research must now shift focus toward lightweight architectures, dynamic complexity control, and hardware-friendly designs Standardization and compatibility are critical as DLF must satisfy stringent engineering requirements like integer-only fixed-point implementation via frameworks like SADL ACKNOWLEDGEMENTS This work was supported by the National Research Foundation of Korea (NRF) grant funded by the Korea Government (MSIT) under Grant NRF-RS-2026-25489154.

Paper summary: Tom: We talked about how this survey organizes deep learning video filtering methods based on integration scheme, coding information utilization, and network design strategy.

Jane: That three-dimensional taxonomy is the main structure they propose for understanding the whole DLF landscape. It’s a way to categorize everything from where you put the filter in the pipeline to what kind of information it uses.

Lu: Basically, dimension one deals with integration scheme within video coding, which controls pipeline latency and bitstream syntax dependency Jane.

Meng: Dimension two is about coding information utilization—how they feed spatial context or temporal context into the network to get better results Tom.

Lalam: And dimension three is about network design strategy, covering things like residual blocks and attention mechanisms, which are structural building blocks for these models Lu.

Tom: The paper claims its main contributions are this systematic taxonomy and the unified system-level analysis across the entire pipeline, focusing on hardware constraints like MAC operations and memory bandwidth Jane.

Jane: It’s important because it moves the conversation from just "this filter works" to "this filter works under these specific hardware limitations," especially for HEVC or VVC systems Lu.

Lu: And they show how different techniques have been applied across various standards in a table, like showing Residual learning on HEVC HM-sixteen point zero and then moving to more complex temporal structure fusion for VTM-three point zero Lalam <ref:2607.16319#pg3>.

Tom: The results of the performance trade-off analysis show a clear evolution towards ultra lightweight architectures that use less than fifty thousand MACs per pixel to satisfy strict hardware constraints Jane.

Meng: That shift toward low MACs is what we’re aiming for in production because that directly addresses the power consumption and memory bandwidth issues we face at scale Tom.

Jane: They also detail the specific challenges for deployment, like needing integer only inference using SADL frameworks and avoiding line buffers Lu.

Lu: The paper also clearly outlines what the next steps should be, pointing towards meta learning or few shot adaptation to let filters handle new video domains with minimal data Jane.

Tom: So, to summarize this part of the paper: it’s a comprehensive system-level analysis that sets up a framework for researchers to systematically design DLF methods around pipeline constraints.

Meng: And for us engineers, it highlights the tight engineering requirements we have to meet, especially concerning integer arithmetic and memory limits Jane.

Lalam: It connects these high-level design choices directly to the practical limitations of the hardware they are designed for Lu.

Conclusion: Tom: So wrapping up this paper, "Deep Learning-based Filtering for Video Coding: A Survey on Architectures, Algorithms, and Complexity Analysis" is essentially a map of where deep learning video filtering is currently headed.

Jane: Right. It shows that DLF has shown real capability in fixing complex non linear artifacts that traditional filters can't handle Lu.

Lu: But the authors make it clear that the primary bottleneck for commercial deployment right now remains computational cost and memory bandwidth Tom.

Meng: They emphasize that while performance is good, you can’t just throw a huge model at a device if it runs out of power or memory Jane.

Lalam: The key findings are that the evolution must shift toward ultra lightweight architectures and dynamic complexity control to satisfy those strict hardware constraints Lu.

Tom: And standardization and compatibility are crucial because DLF has to meet engineering requirements like integer only fixed point implementation through frameworks like SADL Jane.

Jane: It’s about moving from research that just shows a cool result to designs that actually work in the real world for IoT and consumer electronics Lu.

Sunmoon University · Korea Advanced Institute of Science and Technology

cs.CV

Submitted: 2026-07-15

Updated: 2026-11-10

DOI: 10.1109/TCE.2026.3711657

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 80/100

The gist: Deep Learning-based Filtering (DLF) has emerged as a promising solution to mitigate compression artifacts inherent in standards like High Efficiency Video Coding (HEVC/H.265) and Versatile Video…

Key concepts

Integration Scheme
This refers to where the filtering happens within the video processing pipeline. Options include Post-Processing Filtering (PPF) done after decoding, In-Loop Filtering (ILF) embedded during encoding, and Fractional-pixel Interpolation Filtering (FIF). Each scheme has different constraints on latency and complexity.
Coding Information Utilization
This examines how the DL model uses video data features to improve filtering. Key approaches involve using spatial context from a single frame, temporal context by fusing multiple frames to exploit redundancy, and utilizing coding priors like prediction signals or quantization parameters.
Network Design Strategies
These are structural innovations in the neural network architecture. Common building blocks include Residual Blocks and Attention Blocks to model differences between distorted and original signals. Complexity-oriented strategies like Dynamic & Early Exit help reduce computation by exiting the network early for simple samples.

Terminology

Summary

Deep Learning-based Filtering (DLF) has emerged as a promising solution to mitigate compression artifacts inherent in standards like High Efficiency Video Coding (HEVC/H. p. 0265) and Versatile Video Coding (VVC/H. p. 0266), but its deployment is severely constrained by computational complexity, memory bandwidth, and power consumption

Systematic Classification of DLF Methods

The paper proposes a systematic three-dimensional taxonomy classifying methods into (1) Integration Scheme within the Video Coding, (2) Coding Information Utilization, and (3) Network Design Strategy This taxonomy organizes the DLF design space along three axes that each govern a distinct CE deployment constraint Dimension 1 determines pipeline latency and bitstream-syntax dependency, Dimension 2 determines parsing dependency and memory bandwidth, and Dimension 3 determines per-pixel MACs and on-chip storage

Analysis of Integration Schemes

The integration scheme dimension is divided into Post-Processing Filtering (PPF), In-Loop Filtering (ILF), and Fractional-pixel Interpolation Filtering (FIF) PPF operates only at the decoder side as a post-processor, focusing on visual enhancement without affecting the coding loop ILF is embedded within the coding loop, where filtered frames serve as reference frames FIF replaces the interpolation filter in the ME/MC module and is invoked frequently at the block level, making its per-pixel complexity constraints significantly stricter than frame-level filters

Utilization of Coding Information

The second dimension explores how various information sources are utilized to enhance filtering performance Key differentiators include spatial context, temporal context, and utilization of coding priors Spatial context focuses on utilizing spatial information within a single frame to restore textures lost due to quantization Temporal context involves multi-frame fusion strategies that exploit temporal redundancy inherent in video data Utilization of coding priors includes using partition information, prediction signals, and QP/Mode information to guide the network toward artifact-prone regions

Network Design Strategies for Hardware Feasibility

The third dimension discusses the network design strategies and structural innovations employed in StateOf-The-Art (SOTA) DLF models Key building blocks include the Residual Block, Dense Block, Recursive Block, and Attention Block Structural priors such as Residual Learning are standard to model the sparse difference between distorted and original signals Attention mechanisms like Channel Attention explicitly model inter-dependencies between channels, emphasizing feature maps that contain relevant artifact information Complexity-oriented designs include Dynamic & Early Exit strategies where easy samples exit the network at early layers to reduce average computation

Performance and Complexity Trade-offs

The paper presents an extended trade-off analysis comparing HEVC Test Model (HM), VVC Test Model (VTM), and Enhanced Compression Model (ECM) across various coding configurations The evolution shows a paradigm shift towards ultra-lightweight architectures (< 50k MACs/pixel) to satisfy strict hardware constraints Ablation studies show that removing the explicit partitioning map or boundary strength input from the network resulted in negligible performance loss while reducing computational complexity The latest research trends demonstrate a paradigm shift towards ultra-lightweight architectures (< 50k MACs/pixel) to satisfy the strict hardware constraints of emerging standards like JVET NNVC

Standardization and Future Challenges

For DL technologies to be adopted in future standards, they must satisfy stringent engineering requirements such as integer-only fixed-point implementation via frameworks like SADL Strict caps on MACs/pixel and the elimination of line-buffering are critical for minimizing on-chip memory usage Open challenges include real-time inference latency, error propagation in ILF, domain generalization, and NPU-friendly model design Future work should explore meta-learning or few-shot adaptation techniques to allow the filter to adapt to new video domains with minimal data and updates Temporal consistency and error propagation control are key concerns for PPF and ILF, requiring mechanisms like bi-directional prediction or Kalman filtering to suppress flickering The primary challenge is the prohibitive computational cost and memory bandwidth of deep models

Conclusion

DLF is poised to become an indispensable component of next-generation video coding systems with the advancement of DL-specific accelerators (NPUs) Intelligent video coding satisfying high performance, low power, and low latency will soon become a reality in IoT and CE The key findings are that DLFs have demonstrated remarkable capabilities in mitigating complex non-linear artifacts that traditional handcrafted filters fail to address Despite performance gains, computational complexity and memory bandwidth remain the primary bottlenecks for commercial deployment Building upon these foundational heavy models, the natural evolution of current research must now shift focus toward lightweight architectures, dynamic complexity control, and hardware-friendly designs Standardization and compatibility are critical as DLF must satisfy stringent engineering requirements like integer-only fixed-point implementation via frameworks like SADL

ACKNOWLEDGEMENTS

This work was supported by the National Research Foundation of Korea (NRF) grant funded by the Korea Government (MSIT) under Grant NRF-RS-2026-25489154.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Replace conventional filters with lightweight, hardware-friendly DL models that adhere to strict constraints like integer-only inference and bounded MACs/pixel, as suggested by "the latest research trends demonstrate a paradigm shift towards ultra-lightweight architectures (< 50k MACs/pixel) to satisfy the strict hardware constraints of emerging standards like JVET NNVC."

  2. Implement dynamic complexity control where a policy network determines the optimal network topology for each block on-the-fly, which allows the system to adapt its computational path based on input characteristics to optimize complexity without performance loss.

  3. Develop domain generalization capabilities by employing online finetuning, where only bias parameters are updated at the encoder and transmitted, enabling the filter to adapt to new video domains with minimal data and updates.

  4. Integrate spatial-channel mixing blocks into neural network-based in-loop filters to explicitly separate and sequentially apply spatial and channel mixing operations, enhancing chroma filtering performance.

  5. Design architectures that utilize pixel embedding for fractional interpolation in video coding, which can be used to bypass heavy CNN computations, reducing both memory access and computational load compared to traditional methods.

Sources

Related papers