Restoring without Forgetting: Filter-Level Continual Image Restoration via Parameter-Space Integrated Gradients

arXiv:2609.38591 · cs.CV · Submitted 2026-09-29 · Read on arXiv

cs.CV

Submitted: 2026-09-29

Updated: 2026-09-29

Code: https://github.com/funkdub/RwF

License: http://creativecommons.org/licenses/by/4.0/

The gist: Adapting image restoration models to a stream of new tasks without revisiting past data remains challenging due to catastrophic forgetting.

Terminology

Abstract

Adapting image restoration models to a stream of new tasks without revisiting past data remains challenging due to catastrophic forgetting. In this work, we propose Restoring without Forgetting (RwF), a filter-level continual adaptation framework for image restoration built upon a critical observation: task-specific knowledge is centered in a small subset of filters and can be separated from those reconstructing general content. RwF first performs parameter-space integrated gradients attribution to localize degradation-critical filters in a coarse-to-fine manner. It then adapts to new tasks by generating task-specific filters from a filter bank using compact factorized low-rank transformations, further augmented with cross-task attention and prototypical contrastive learning, and lastly assembles them back only at localized positions. Experiments on six restoration tasks show that RwF effectively avoids forgetting and achieves competitive restoration quality against all-in-one methods that have full data access, and outperforms LoRA-style adaptation with about 10 times fewer additional parameters. Code is available at https://github.com/funkdub/Restoring-without-Forgetting.

Sources

Related papers