FreeTransformSR: Efficient Lightweight Image Super-Resolution via Free Low-Rank Learnable Transform

arXiv:2609.05912 · cs.CV · Submitted 2026-09-05 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "FreeTransformSR: Efficient Lightweight Image Super-Resolution via Free Low-Rank Learnable Transform".

Jane: Here's the gist of the paper. Single Image Super-Resolution (SISR) aims to reconstruct high-resolution images from low-resolution inputs The paper proposes FreeTransformSR, a novel lightweight super-resolution network based on a channel-wise free low-rank learnable transform This method achieves competitive PSNR/SSIM performance with significantly fewer parameters and FLOPs.

Tom: Network Architecture Overview The overall architecture of FreeTransformSR is illustrated in Fig.

Jane: 1, which mainly consists of three modules: shallow feature extraction, deep feature extraction, and image reconstruction Given a low-resolution input image I L R ∈ R H×W×3, we first apply a 3×3 convolution for shallow feature extraction.

Tom: The shallow features F 0 are then sequentially passed through N residual groups (RGs) for deep feature extraction.

Title and authors: Jane: So, to summarize the core idea of FreeTransformSR: it uses a learnable channel-wise free low-rank transform combined with adaptive fusion and intensity modulation to get better super resolution.

The paper's summary: Tom: Now that we’ve covered the basics, let’s look at the core summary of FreeTransformSR. It essentially boils down to this: they are proposing a network architecture that uses three main modules—shallow feature extraction, deep feature extraction, and image reconstruction.

Jane: So it starts with a three times three convolution for shallow features, which are then passed through N residual groups for deep features, where each residual group has learned transform blocks and soft complexity adaptive blocks.

Lu: And at the end of each residual group, they apply a one times one convolution and an Efficient Channel Attention module to aggregate and recalibrate those features <ref:2609.05912#pg3>.

Meng: It sounds like a deep structure with attention integrated throughout the feature extraction part, which is more complex than just stacking standard residual blocks. What’s the specific benefit here?

Tom: The benefit is that this whole setup is designed to be lightweight while still achieving competitive performance, which they emphasize by focusing on minimizing parameters and FLOPs.

Jane: And the key innovation they highlight, as described in the abstract, is that their core components are built around a channel-wise free low-rank learnable transform.

Lu: That transform learns a pair of trainable 2D transformation matrices for each feature channel independently, which allows it to map windows from the spatial domain to the transform domain.

Meng: So instead of one fixed set of matrices for all features, you get a different set per channel, which sounds like it could offer much finer control over how information is transformed.

Tom: Right. And they also add an adaptive intensity modulation module that extracts global responses from input features to generate a sample-level scaling coefficient S.

Jane: That means the network can dynamically adjust the response strength of its transform-domain processing at the sample level based on what it sees in the input features.

Lu: So, putting it together, this is a system that combines data driven transform learning with adaptive feature fusion and sample-level intensity control to tackle super resolution.

Meng: That sounds like a lot of mechanisms working together to keep things efficient and accurate, which is exactly what we need when shipping models to production.

Tom: Exactly. So, moving on, where do we see the biggest practical impact? It seems like they’re focusing on making sure every component serves a specific purpose in this efficient pipeline.

Jane: The focus is really on demonstrating that this combination of lightweight components can outperform heavy methods without needing a huge computational budget for training or inference.

Lu: It shows that representation learning can be highly flexible, allowing the network to learn the right mathematical basis functions for the task at hand.

The paper's improvements: Tom: Let’s dig into those specific improvements they list in FreeTransformSR, because it’s not just a general idea; it’s these distinct mechanisms that are what make it work.

Jane: First improvement is that the learnable channel-wise free low-rank transform allows for data driven basis function evolution, distinguishing it from frequency domain methods that rely on predefined fixed basis functions.

Lu: That means the network learns a pair of 2D transformation matrices for each feature channel independently, offering stronger data adaptability compared to fixed-basis methods.

Meng: So, instead of one fixed set of matrices for all features, you get a different set per channel, which sounds like it could offer much finer control over how information is transformed.

Tom: And they also introduce a local feature modulation branch that complements transform-domain processing with depthwise convolution to help recover high-frequency detail.

Jane: That design enables local details to modulate global features by applying depthwise separable convolution to a branch that has not undergone transform-domain processing, and then multiplying these local features element wise with the transform domain recovered features.

Lu: This allows local details to modulate global features, enabling the network to maintain global consistency while preserving high-frequency details.

Meng: That’s smart because it tackles the trade off between keeping things globally coherent and getting those sharp details back.

Tom: And then we have the soft complexity adaptive module that dynamically fuses the outputs of local convolution and window self attention branches through a lightweight gating network, which adaptively adjusts the fusion ratio based on regional texture characteristics.

Jane: It means it’s not making a hard binary choice between smooth and textured regions; it’s selecting an appropriate fusion ratio according to regional image characteristics.

Lu: That allows two operations with different inductive biases to work synergistically, which is a key design choice here for combining local convolution and window self-attention.

Meng: So it’s not just stacking layers; it’s intelligently deciding how much to trust each part of the feature extraction pipeline based on what the image region looks like.

Tom: And finally, they have that adaptive intensity modulation strategy that dynamically adjusts the response strength of transform-domain processing at the sample level based on input features.

Jane: This means it allows the network to autonomously determine the degree of amplification or suppression for different inputs based on what it sees in the input features.

Lu: So, putting these improvements together, you have a system that’s highly flexible due to data driven transform learning and adaptive feature fusion and sample-level intensity control.

Meng: It sounds like they’ve really worked hard on making sure every single piece of this mechanism is designed to serve a specific purpose in this efficient pipeline.

Conclusion: Tom: So, to wrap up the FreeTransformSR paper: it proposes a method with learnable channel-wise transforms, adaptive fusion of local and attention branches based on texture, and intensity modulation to achieve efficient image super resolution.

Jane: It’s a super resolution network that uses a channel-wise free low-rank learnable transform to reconstruct high resolution images from low resolution inputs.

Meng: For deployment purposes, the parameter count is huge, but I need more data on how robust this system is when deployed on real traffic feeds versus clean test sets.

Tom: The key thing to remember here is that they achieved competitive PSNR and SSIM scores with significantly fewer parameters and FLOPs compared to competing methods.

Jane: So, the main implication for us is that we have a really efficient tool now for deploying super resolution in resource constrained scenarios.

Lu: It’s a solid approach because it combines different inductive biases, allowing the network to learn the right mathematical basis functions for the task at hand.

Meng: I still need more data on how robust this system is when deployed on real traffic feeds versus clean test sets.

Tom: The final piece of the FreeTransformSR paper is that they achieved competitive PSNR and SSIM scores with significantly fewer parameters and FLOPs compared to competing methods.

Jane: So, to wrap up: FreeTransformSR provides an efficient and practical solution for super-resolution deployment in resource constrained scenarios.

Chinese Academy of Sciences Changchun Institute of Optics Fine Mechanics and Physics · University of the Chinese Academy of Sciences

cs.CV

Submitted: 2026-09-05

Updated: 2026-10-29

Comments: 20 pages, 5 figures

Code: https://github.com/HJiLi/FreeTransformSR

License: http://creativecommons.org/licenses/by-nc-nd/4.0/

Importance score: 79/100

The gist: Single Image Super-Resolution (SISR) aims to reconstruct high-resolution images from low-resolution inputs The paper proposes FreeTransformSR, a novel lightweight super-resolution network based on a

Key concepts

Learnable Channel-wise Free Low-Rank Transform
This mechanism learns independent 2D transformation matrices for each feature channel. Unlike fixed frequency domain methods, this allows the network to adapt its basis functions based on the input data, enabling a more flexible and data-driven way to transform spatial features.
Soft Complexity Adaptive Module
This module dynamically fuses information from two different branches (local convolution and window self-attention) using a lightweight gating network. This gate learns to select the optimal fusion ratio based on the specific characteristics of the image region, avoiding rigid binary choices between different processing styles.
Adaptive Intensity Modulation
This strategy adjusts how strongly transform-domain processing is applied at each sample point based on input features. It allows the network to autonomously decide whether to amplify or suppress specific parts of the transformation output, tailoring the response strength dynamically.
Local Feature Modulation Branch
This branch uses a depthwise separable convolution to enhance high-frequency details without applying transform-domain processing. These local features are then multiplied element-wise with the features recovered from the transform domain, ensuring that fine details modulate global features while maintaining overall image consistency.

Terminology

Summary

Single Image Super-Resolution (SISR) aims to reconstruct high-resolution images from low-resolution inputs The paper proposes FreeTransformSR, a novel lightweight super-resolution network based on a channel-wise free low-rank learnable transform This method achieves competitive PSNR/SSIM performance with significantly fewer parameters and FLOPs.

Network Architecture Overview

The overall architecture of FreeTransformSR is illustrated in Fig. 1, which mainly consists of three modules: shallow feature extraction, deep feature extraction, and image reconstruction Given a low-resolution input image I L R ∈ R H×W×3, we first apply a 3×3 convolution for shallow feature extraction. The shallow features F 0 are then sequentially passed through N residual groups (RGs) for deep feature extraction. Each residual group consists of several basic blocks, which are categorized into two types: Learned Transform Blocks (LTB) and Soft Complexity Adaptive Blocks (SCAB). At the end of each residual group, a 1×1 convolution and an Efficient Channel Attention (ECA) [29] module are applied for feature aggregation and recalibration.

Core Transform Mechanism

The core innovations of this network are threefold First, at the feature transformation level, we design a data-driven channel-wise learnable transform. This transform learns a pair of trainable 2D transformation matrices (row transform and column transform) for each feature channel independently. The forward transform maps each window block from the spatial domain to the transform domain. Additionally, we introduce an adaptive intensity modulation module that extracts global responses from the input features and generates a sample-level scaling coefficient S.

Adaptive Feature Fusion

Second, at the feature fusion level, we design a soft complexity adaptive module that dynamically fuses the outputs of a local convolution branch and a window self-attention branch through a lightweight gating network. The gate computes a continuous fusion coefficient based on input features, enabling the model to adaptively select an appropriate fusion ratio according to regional image characteristics. This module allows two operations with different inductive biases to work synergistically.

Training Objectives and Loss Function

To simultaneously constrain the quality of reconstructed images in both the spatial and frequency domains, we adopt a weighted combination of L1 pixel loss and Fourier space loss as the total loss function. L1 Pixel Loss measures the per-pixel absolute error between the reconstructed image I S R and the highresolution ground truth I H R. The Fourier Space Loss constrains reconstruction quality in the frequency domain. The total loss function is the weighted sum of these two losses.

Experimental Validation and Efficiency

Experiments on five benchmark datasets demonstrate that FreeTransformSR achieves competitive PSNR/SSIM performance with significantly fewer parameters and FLOPs. Specifically, FreeTransformSR achieves 32.41 dB on BSD100 ×2 and 27.00 dB on Urban100 ×4 with only 653K parameters, while delivering faster inference speed than competing methods. The adaptive intensity modulation strategy dynamically adjusts the response strength of transform-domain processing at the sample level based on input features. Ablation studies validate the effectiveness of each core module and confirm that the 2+2 configuration offers the optimal trade-off between accuracy and efficiency. The transposed back-mapping strategy maintains reconstruction performance while avoiding extra computational burden. The model achieves the fastest inference throughput at the ×2 scale (10.66 imgs/s) and the second-fastest at the ×4 scale (27.32 imgs/s).

Future Directions

Future work will explore extending the framework to complex real-world degradation scenarios and larger-scale training data, as well as investigating adaptive window mechanisms to better capture multi-scale structural information. FreeTransformSR provides an efficient and practical solution for super-resolution deployment in resource-constrained scenarios. The paper's main contributions are summarized as follows: we propose a learnable channel-wise free low-rank transform that achieves adaptive feature modulation in a data-driven manner, we design a soft complexity adaptive gating structure that adaptively fuses local convolution and window self-attention according to regional texture characteristics, and we introduce adaptive intensity modulation to dynamically adjust transform-domain response strength at the sample level. The authors declare no conflict of interest. The work was supported in part by the National Natural Science Foundation of China under Grant 62005266, and in part by the Youth Innovation Promotion Association of the Chinese Academy of Sciences under Grant 2022219. The authors are Hongji Li a,b and Yunhui Li a, with email liyunhui@ciomp.ac.cn.

Improvements for AI systems

  1. Learnable channel-wise free low-rank transform allows for data-driven basis function evolution, distinguishing it from frequency domain methods that rely on predefined fixed basis functions. This enables adaptive feature modulation by learning a pair of 2D transformation matrices for each feature channel independently, offering stronger data adaptability compared to fixed-basis methods.

  2. Soft complexity adaptive module dynamically fuses the outputs of local convolution and window self-attention branches through a lightweight gating network, adaptively selecting an appropriate fusion ratio based on regional texture characteristics. This allows the model to adaptively select an appropriate fusion ratio according to regional image characteristics, rather than making a hard binary choice between smooth and textured regions.

  3. Adaptive intensity modulation strategy dynamically adjusts the response strength of transform-domain processing at the sample level based on input features, enabling the network to autonomously determine the degree of amplification or suppression for different inputs based on the statistical characteristics of the training data.

  4. The local feature modulation branch enhances high-frequency detail recovery by applying depthwise separable convolution to a branch that has not undergone transform-domain processing, and then multiplying these local features element-wise with the transform-domain recovered features. This design enables local details to modulate global features, enabling the network to maintain global consistency while preserving high-frequency details.

  5. The network can be deployed in resource-constrained scenarios due to its competitive PSNR/SSIM performance with significantly fewer parameters and FLOPs compared to competing methods, such as achieving 32.41 dB on BSD100 ×2 with only 653K parameters.

Abstract

Single image super-resolution aims to reconstruct high-resolution images from low-resolution inputs. This paper proposes FreeTransformSR, a novel lightweight super-resolution network based on a channel-wise free low-rank learnable transform. The transform learns task-adaptive basis functions in a data-driven manner, enabling adaptive feature modulation with minimal parameter overhead. To further enhance high-frequency detail recovery, we introduce a local feature modulation branch that complements transform-domain processing with depthwise convolution. In addition, a soft complexity adaptive module dynamically fuses the outputs of local convolution and window self-attention branches through a lightweight gating network, adaptively adjusting the fusion ratio based on regional texture characteristics. An adaptive intensity modulation strategy is also incorporated to adjust transform-domain response strength at the sample level, enabling the network to dynamically adjust processing intensity according to input features. Extensive experiments on five benchmark datasets demonstrate that FreeTransformSR achieves competitive PSNR/SSIM performance with significantly fewer parameters and FLOPs. Specifically, FreeTransformSR achieves 32.41 dB on BSD100 x2 and 27.00 dB on Urban100 x4 with only 595K parameters, while delivering faster inference speed than competing methods, making it well-suited for deployment in resource-constrained scenarios. Source code is available at: https://github.com/HJiLi/FreeTransformSR.

Sources

Related papers