Degradation-Aware Unfolding Half-Shuffle Transformer for Spectral Compressive Imaging
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Electrical Engineering. The day in electrical engineering: signals, systems and the hardware that carries them.
Rosa: We're looking at the paper: "Degradation-Aware Unfolding Half-Shuffle Transformer for Spectral Compressive Imaging".
Title and authors: Rosa: So we're looking at this paper now called Degradation-Aware Unfolding Half-Shuffle Transformer for Spectral Compressive Imaging. It’s all about taking hyperspectral images from coded aperture snapshot spectral imaging systems and getting them back from a compressed measurement.
Dev: Yeah, it tackles the two main problems with deep unfolding methods that we talked about before, which is that they don't figure out the degradation patterns or how ill-posed the problem is to guide their iterative learning.
Taro: And another big issue they point out is that most of these deep unfolding methods are built around convolutional neural networks, and those struggle to capture long-range dependencies in the data.
Rosa: So what this paper proposes is a principled Degradation-Aware Unfolding Framework, which uses maximum a posteriori theory to estimate some important parameters from the compressed image and the physical mask that was used for modulation.
Dev: It's interesting because these estimated parameters, which they call alpha and beta, then guide every single iteration of the unfolding process by adaptively scaling the linear projection and giving noise level information to the denoising network.
Taro: That sounds like they are trying to make the iterative learning smarter by feeding it context about how bad the measurement is.
Rosa: Right, and on top of that, they customize a novel Half-Shuffle Transformer, which acts as the denoiser prior in each step. This transformer is designed to capture both local contextual information and those long-range dependencies that were missing in earlier CNN approaches.
Dev: And this transformer uses something called a Half Shuffle Multihead Self Attention mechanism that combines the benefits of local window-based attention with non-local attention, which they claim is computationally cheaper than a global self attention mechanism.
Taro: So if I'm thinking about real-world performance, what does this mean for how robust the reconstruction is when we move it out of the lab and into messy environments?
Rosa: Well, the paper claims that by plugging this Half Shuffle Transformer into their Degradation-Aware Unfolding Framework, they establish a new method called Degradation-Aware Unfolding Half-Shuffle Transformer.
Dev: And they say this new method surpasses state of the art deep unfolding methods by over four decibels in performance. That's a pretty significant jump when you're talking about reconstruction quality.
Taro: So, what are the specific details on how much better it is compared to those older methods we mentioned, like DNU or ADMM?
Rosa: The break-down ablation study showed that by combining the Degradation Aware Unfolding Framework and the Half Shuffle Multihead Self Attention mechanism together, they got a gain of four point one six decibels compared to Baseline one.
Dev: And when they compared the Half Shuffle Multihead Self Attention against other self attention mechanisms, it yielded a gain of one point two six decibels over G MSA, SW MSA and S MSA. That shows that their specific mechanism is quite effective in this context.
Taro: So, what about the overall comparison to other unfolding frameworks like DNU or ADMM?
Rosa: The comparison showed that the Degradation Aware Unfolding Framework significantly outperforms previous methods like DNU by two point five nine decibels, and ADMM by one point six nine decibels and GAP by one point six three decibels while only adding a small increase in parameters and flops.
Dev: It seems like they're achieving this performance boost without a massive increase in computational load, which is something engineers always look for.
Taro: So, to wrap up the main points of this Degradation-Aware Unfolding Half-Shuffle Transformer for Spectral Compressive Imaging paper, what are the biggest implications for us working on these systems?
Rosa: The main implication is that they solved those two core issues: estimating degradation parameters to guide the iterative learning and using a Transformer instead of just CNNs to handle long-range dependencies.
Dev: It moves deep unfolding methods forward by making them more principled, which means they aren't just black box mappings anymore; they are informed by the actual physics of the CASSI system.
Taro: And for people who only listen to this show, it means we might be able to get much better, more robust reconstruction quality in real hyperspectral imaging applications when dealing with noisy data.
Rosa: Exactly. The Degradation-Aware Unfolding Half-Shuffle Transformer for Spectral Compressive Imaging is a lot of work that successfully remedies those two main hurdles while keeping the computational cost reasonable compared to what we see in other deep learning models.
Dev: It's a solid result, but we should keep an eye on how they handle the noise distortion in real HSI reconstruction, because that's where these methods really need to shine for practical use.
Taro: I just think it’s encouraging to see them integrate this kind of principled parameter estimation into iterative schemes like this.
Rosa: It really is, and it sets a new direction for how we approach these inverse problems in spectral imaging.
The paper's summary: Rosa: So, to wrap up this paper, they've put together this whole system called Degradation Aware Unfolding Half Shuffle Transformer to fix two big problems in hyperspectral image reconstruction from coded aperture snapshot spectral imaging systems.
Dev: Right, so basically they took these two nagging issues we talked about—not knowing the degradation patterns and relying too much on CNNs for long-range dependencies—and they tackled them head-on.
Rosa: Exactly. They introduced this Degradation Aware Unfolding Framework, which uses math from MAP theory to estimate parameters directly from the degraded measurement and the physical mask used in the modulation setup.
Dev: Those estimated parameters, alpha and beta, are then fed into every step of the iterative unfolding process. This allows them to adaptively scale how they project data and even tells the denoising network what kind of noise level it's dealing with at each iteration.
Rosa: And for that denoising part, they use this new Half Shuffle Transformer. It’s a Transformer designed to handle both local context and long-range dependencies simultaneously, but they made it computationally cheap enough that it doesn't get in the way of the real-time loop.
Dev: The mechanism they use is a Half Shuffle Multihead Self Attention. It mixes local window attention with non-local attention so it can capture those far-reaching relationships without paying the heavy computational price of a global self attention.
Rosa: The final system, Degradation Aware Unfolding Half Shuffle Transformer, ends up outperforming state of the art methods by over four decibels in performance and still keeps the computational cost relatively low.
Dev: It sounds like they managed to get a big jump in quality without needing a massive increase in processing time or memory usage, which is what engineers always want to see.
Rosa: The real impact here is that this isn't just another black box mapping function; it’s an iterative process that's actually informed by the physical hardware and the specific noise characteristics of the CASSI system.
Dev: So, what this means for us down on the ground is that when we're trying to build systems for real hyperspectral imaging, we can now expect reconstruction quality to be much more robust against noise distortion in live data.
Rosa: It’s a big step toward making these systems reliable enough for actual deployment where the world isn't perfectly calibrated.
The paper's improvements: Taro: So we've seen how they built this system, now let's talk about what makes it actually better than what came before.
Rosa: Yeah, so their main improvements center on two things: first, that parameter estimation part from the Degradation Aware Unfolding Framework.
Dev: Right. They showed that by using these estimated parameters to control the iterative learning—scaling the linear projection and giving noise level info—they get a real adaptive advantage in every step.
Taro: That's interesting because it means if the CASSI system is really noisy or has weird degradation, the unfolding process can actually adjust its own behavior instead of just running on a fixed schedule.
Rosa: Exactly. And then they have the Half Shuffle Transformer itself, which is their second big improvement.
Dev: The paper highlights that this Transformer doesn't just use standard attention; it uses a specific mechanism called the Half Shuffle Multihead Self Attention to combine local and non-local context efficiently.
Taro: So what does that actually mean for real-world failure modes? It means the system can track things across long distances in the image without needing a global attention mechanism that would be too slow or too heavy.
Rosa: Exactly. The authors claim this combination—the framework and the HST—gets them over four decibels better than state of the art methods while keeping the computational load much cheaper.
Dev: That's a huge win for real-time loops, Rosa; a four decibel gain with lower computational costs is pretty compelling when you're thinking about deployment.
Taro: It moves us closer to having these complex reconstruction tools that can handle messy data gracefully and recover well when the input isn't perfect.
Rosa: So what this means for us as people who build these things is that we can stop just relying on fixed architectures and start building iterative systems that are informed by the specific physics of the sensing hardware they are working with.
Dev: It really changes how we design these inverse problems; it’s not just about fitting a model anymore, it's about letting the system learn how to handle the degradation itself through those estimated parameters.
Taro: And for anyone listening who is interested in autonomy, this shows that you can integrate powerful sequence modeling like Transformers into these reconstruction tasks without immediately hitting massive computational walls.
Rosa: It sets a new direction for how we approach these inverse problems; instead of just trying to get the best final image, we're focusing on making the entire iterative process more intelligent and adaptive.
Conclusion: Rosa: So to wrap up this Degradation Aware Unfolding Half Shuffle Transformer paper, they successfully tackled those two main issues in hyperspectral image reconstruction from CASSI systems.
Dev: Right, so they showed that by estimating parameters from the compressed measurement and physical mask, they can adaptively control the iterative learning process through scaling and noise level information.
Taro: It means when we run these systems outside the lab, even if things get noisy or degraded in unexpected ways, this framework is designed to recover more gracefully.
Rosa: And that's paired with their Half Shuffle Transformer, which uses a novel attention mechanism to capture long-range dependencies while keeping the computational cost down.
Dev: The results are pretty strong; they surpass state of the art methods by over four decibels while maintaining a lower memory footprint than some existing deep unfolding methods.
Taro: I just think having a principled way to estimate that degradation degree is what makes this useful for real-world autonomy, because you need to know if your perception system is dealing with something genuinely noisy or just an artifact of the measurement setup.
Rosa: It definitely moves us past just plugging in standard CNNs and starts building systems that are actually aware of their physical environment.
Dev: We should keep an eye on how they handle those real-world noise distortions because that’s where we need to prove this works reliably in a live loop setting.
Taro: I'm curious to see if they can extend this framework to handle even more complex, dynamic environmental changes than what’s shown in these initial trials.
Shenzhen International Graduate School, Tsinghua University · Shenzhen Institute of Future Media Technology · Westlake University · §ETH Zurich · §University of Würzburg
eess.IV, cs.CV
Submitted: 2022-05-20
Updated: 2026-11-17
Comments: NeurIPS 2022; The first Transformer-based deep unfolding method for spectral compressive imaging
Code: https://github.com/caiyuanhao1998/MST
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Importance score: 95/100
The gist: In coded aperture snapshot spectral imaging (CASSI) systems, hyperspectral image (HSI) reconstruction methods are employed to recover the spatialspectral signal from a compressed measurement Among…
Key concepts
- Degradation-Aware Unfolding Framework (DAUF)
- This framework uses Maximum A Posteriori (MAP) theory to implicitly estimate parameters from the compressed measurement and physical mask. These estimated parameters capture crucial information about how the CASSI system degrades and how ill-posed the problem is, allowing each iteration of the reconstruction to adapt its scaling and noise estimation accordingly.
- Half-Shuffle Transformer (HST)
- The HST acts as the denoiser prior in each step. It uses a Half-Shuffle Multihead Self-Attention (HS-MSA) mechanism that combines local window attention for spatial context with non-local attention to model long-range dependencies. This allows it to capture both nearby details and broader relationships in the image data.
- Half-Shuffle Multihead Self-Attention (HS-MSA)
- This is the core mechanism of the HST. It merges two attention branches: one focusing on local context within specific windows, and another modeling non-local dependencies across the entire image. This hybrid approach captures both fine details and global relationships efficiently, making it more effective than standard global self-attention.
- Deep Unfolding Methods
- These are multi-stage neural network architectures used for reconstructing hyperspectral images from compressed measurements. They work iteratively, mapping the compressed measurement into the full HSI cube over several steps. The proposed method improves upon these by adding parameter estimation and a more advanced attention mechanism.
Terminology
Summary
In coded aperture snapshot spectral imaging (CASSI) systems, hyperspectral image (HSI) reconstruction methods are employed to recover the spatialspectral signal from a compressed measurement Among these algorithms, deep unfolding methods demonstrate promising performance but suffer from two issues Firstly, they do not estimate the degradation patterns and ill-posedness degree from CASSI to guide the iterative learning. p. 0 Secondly, they are mainly CNN-based, showing limitations in capturing long-range dependencies. p. 0 This paper proposes a principled Degradation-Aware Unfolding Framework (DAUF) and a novel Half-Shuffle Transformer (HST) to establish the first Transformer-based deep unfolding method, Degradation-Aware Unfolding Half-Shuffle Transformer (DAUHST), for HSI reconstruction. p. 0
Pruning redundant reasoning tokens
The paper introduces the Degradation-Aware Unfolding Framework (DAUF) based on MAP theory to implicitly estimate informative parameters from the degraded compressed measurement and the physical mask used in the modulation <ref:2205.10102#pg5>. This framework adaptsively scales the linear projection and provides noise level information for the denoising network in each iteration by feeding estimated parameters, which capture key cues of CASSI degradation patterns and ill-posedness degree <ref:2205. p. 110102#pg3>. The iterative scheme is formulated as (α, β) = E(y, Φ), xk+1 = P(y, zk, αk+1, Φ), zk+1 = D(xk+1, βk+1) <ref:2205. p. 310102#pg3>.
How the training signal is built
The DAUF framework minimizes an energy function involving a data fidelity term and an image prior term to estimate the original HSI signal ˆx = arg min x 1/2 y − Φx2 + τR(x) <ref:2205.10102#pg4>. This optimization is solved using the half-quadratic splitting (HQS) algorithm, which decouples x and z into two iterative sub-problems to obtain closed-form solutions for xk+1 and zk+1 <ref:2205. p. 210102#pg5>. The penalty parameter µ controls the convergence and output of each iteration, and it is set as a series of iteration-specific parameters denoted as µk <ref:2205. p. 310102#pg5>.
Where the gains come from
The proposed Half-Shuffle Transformer (HST) is designed to serve as the denoiser prior (D network) in each iteration, overcoming the limitations of CNNs by jointly extracting local contextual information and modeling non-local dependencies <ref:2205.10102#pg6>. The HST utilizes a Half-Shuffle Multi-head Self-Attention (HS-MSA) mechanism which combines local branch and non-local branch to capture both local contents and long-range dependencies while being computationally cheaper than global MSA <ref:2205. p. 110102#pg7>. The joint exploitation of the estimated parameters α and β in the iterative learning leads to significant performance improvements, with DAUHST surpassing state-of-the-art (SOTA) deep unfolding methods by over 4 dB <ref:2205. p. 110102#pg3>.
Ablation study results
The break-down ablation study confirms the effectiveness of the components, showing that exploiting both DAUF and HS-MSA jointly yields a gain of 4.16 dB compared to Baseline-1 <ref:2205. p. 010102#pg10>. Furthermore, comparing HS-MSA against other self-attention mechanisms reveals that it yields the most significant improvement of 1. p. 726 dB over G-MSA, SW-MSA, and S-MSA <ref:2205. p. 110102#pg10>. The comparison of unfolding frameworks shows that DAUF significantly outperforms previous methods like DNU, ADMM, and GAP by 2. p. 759, 1.69, and 1.63 dB while adding only a small increase in parameters and FLOPS <ref:2205. p. 710102#pg9>. The analysis of estimated parameters α and β demonstrates that they are beneficial for the linear projection and denoising network of deep unfolding methods <ref:2205. p. 810102#pg5>.
Conclusion
The paper successfully remedies the two main issues of previous deep unfolding methods, namely the lack of parameter estimation from CASSI system to direct iterative learning and their reliance on CNNs limiting long-range dependencies <ref:2205.10102#pg10>. By formulating DAUF to estimate parameters and proposing HST as a novel Transformer, the authors derive DAUHST, which outperforms SOTA methods while requiring cheaper computational and memory costs <ref:2205. p. 810102#pg3>. The comprehensive experiments validate that DAUHST is more robust to noise distortion in real HSI reconstruction <ref:2205. p. 710102#pg9>.
--- Page 1 ---
Degradation-Aware Unfolding Half-Shuffle Transformer for Spectral Compressive Imaging
Yuanhao Cai 1,2,∗
, Jing Lin 1,2,∗
, Haoqian Wang 1,2,†
, Xin Yuan 3
Henghui Ding 4
Yulun Zhang 4 Radu Timofte 4,5 Luc Van Gool 4 <ref:2205.10102#pg2>.
Abstract In coded aperture snapshot spectral compressive imaging (CASSI) systems, hyperspectral image (HSI) reconstruction methods are employed to recover the spatialspectral signal from a compressed measurement <ref:2205. p. 010102#pg3>. Among these algorithms, deep unfolding methods demonstrate promising performance but suffer from two issues <ref:2205. p. 010102#pg3>. Firstly, they do not estimate the degradation patterns and ill-posedness degree from CASSI to guide the iterative learning <ref:2205. p. 010102#pg3>. Secondly, they are mainly CNN-based, showing limitations in capturing long-range dependencies <ref:2205. p. 010102#pg3>. In this paper, we propose a principled Degradation-Aware Unfolding Framework (DAUF) that estimates parameters from the compressed image and physical mask, and then uses these parameters to control each iteration <ref:2205. p. 010102#pg3>. Moreover, we customize a novel Half-Shuffle Transformer (HST) that simultaneously captures local contents and non-local dependencies <ref:2205. p. 010102#pg3>. By plugging HST into DAUF, we establish the first Transformer-based deep unfolding method, Degradation-Aware Unfolding Half-Shuffle Transformer (DAUHST), for HSI reconstruction <ref:2205. p. 010102#pg3>. Experiments show that DAUHST surpasses state-of-the-art methods while requiring cheaper computational and memory costs <ref:2205. p. 010102#pg3>.
Introduction Hyperspectral images (HSIs) have more spectral bands than normal RGB images to store more detailed information <ref:2205. p. 010102#pg2>. Thus, HSIs are widely applied in image recognition [1, 2, 3], object detection [4, 5, 6], tracking [7, 8, 9], medical image processing [10, 11, 12], remote sensing [13-16]. p. 0 To obtain HSIs traditional imaging systems use spectrometers to scan the scenes along the spectral or spatial dimensions usually requiring a long time <ref:2205. p. 0 p. 010102#pg3>. Recently snapshot compressive imaging (SCI) systems have been developed to capture HSIs at video rate <ref:2205. p. 010102#pg3>. Among these SCI systems, coded aperture snapshot spectral imaging (CASSI) stands out for its impressive performance <ref:2205. p. 0 p. 010102#pg4>. CASSI uses a coded aperture and a disperser to modulate the HSI signal at different wavelengths and then mixes all modulated signal to generate a 2D compressed measurement <ref:2205. p. 0 p. 010102#pg4>. Subsequently, HSI restoration methods are employed to solve the CASSI inverse problem i. p. 0 p. 0e., restore the HSIs from the measurement <ref:2205. p. 0 p. 010102#pg4>. These methods are divided into four categories (i) Model-based methods [18, 26-30] rely on hand-crafted image priors like total variation or sparsity <ref:2205. p. 010102#pg4>. (ii) Plug-and-play (PnP) algorithms [33, 34] plug pre-trained denoising networks into traditional model-based methods <ref:2205. p. 1 p. 110102#pg5>. (iii) End-to-end (E2E) algorithms employ a powerful model usually a convolutional neural network to learn the E2E mapping function from measurement to desired HSIs <ref:2205. p. 110102#pg6>. (iv) Deep unfolding methods [43-49] adopt a multi-stage network to map the measurement into the HSI cube <ref:2205. p. 1 p. 110102#pg6>. Existing deep unfolding algorithms suffer from two issues (a) the iterative learning is highly related to the CASSI system, but current unfolding methods do not estimate CASSI degradation patterns and ill-posedness degree <ref:2205. p. 1 p. 110102#pg6>(b) existing deep unfolding methods are mainly CNN-based, showing limitations in capturing non-local self-similarity and long-range dependencies <ref:2205. p. 1 p. 110102#pg6>.
--- Page 2 ---
Hyperspectral images (HSIs) have more spectral bands than normal RGB images to store more detailed information <ref:2205.10102#pg2>. Thus, HSIs are widely applied in image recognition [1, 2, 3], object detection [4, 5, 6], tracking [7-9], medical image processing [10-12], remote sensing [13-16]. p. 0 To obtain HSIs traditional imaging systems use spectrometers to scan the scenes along the spectral or spatial dimensions usually requiring a long time <ref:2205.10102#pg3>. These imaging systems fail to capture dynamic objects <ref:2205. p. 010102#pg3>. Recently snapshot compressive imaging (SCI) systems have been developed to capture HSIs at video rate [17-23]. p. 0 Among these SCI systems, coded aperture snapshot spectral imaging (CASSI) stands out for its impressive performance <ref:2205.10102#pg4>. CASSI uses a coded aperture and a disperser to modulate the HSI signal at different wavelengths and then mixes all modulated signal to generate a 2D compressed measurement <ref:2205.10102#pg4>. Subsequently, HSI restoration methods are employed to solve the CASSI inverse problem i.e., restore the HSIs from the measurement <ref:2205.10102#pg4>. These methods are divided into four categories (i) Model-based methods [18-30] rely on hand-crafted image priors, e. p. 0g., total variation or sparsity <ref:2205. p. 010102#pg4>. (ii) Plug-and-play (PnP) algorithms [33, 34] plug pre-trained denoising networks into traditional model-based methods <ref:2205.10102#pg5>. (iii) End-to-end (E2E) algorithms employ a powerful model usually a convolutional neural network to learn the E2E mapping function from a measurement to the desired HSIs <ref:2205. p. 110102#pg6>. (iv) Deep unfolding methods [43-49] adopt a multi-stage network to map the measurement into the HSI cube <ref:2205.10102#pg6>. Existing deep unfolding algorithms suffer from two issues (a) the iterative learning is highly related to the CASSI system, but current unfolding methods do not estimate CASSI degradation patterns and ill-posedness degree <ref:2205.10102#pg6>(b) existing deep unfolding methods are mainly CNN-based, showing limitations in capturing non-local self-similarity and long-range dependencies <ref:2205.10102#pg6>. Recently the emerging Transformer [50] has provided a solution to tackle the drawbacks of CNN <ref:2205. p. 110102#pg7>. Due to its strong capability in modeling the interactions of non-local spatial regions, Transformer has been widely applied in image classification [51-57], object detection [58, 60], semantic segmentation [51, 64], and image restoration [76-82]. p. 1 Yet the use of Transformer is confronted with two main issues (a) the computational complexity of global Transformer is quadratic to the spatial dimensions <ref:2205. p. 110102#pg8>. (b) the receptive fields of local Transformer are limited within position-specific windows <ref:2205. p. 110102#pg8>. As a result, some tokens with highly-related contents can not match each other when computing the self-attention <ref:2205. p. 110102#pg8>.
--- Page 3 ---
To address the above problems, in this paper, we firstly formulate a principled Degradation-Aware Unfolding Framework (DAUF) based on maximum a posteriori (MAP) theory for HSI reconstruction <ref:2205.10102#pg2>. Different from previous deep unfolding methods, our DAUF implicitly estimates informative parameters from the degraded compressed measurement and the physical mask used in the modulation <ref:2205. p. 110102#pg3>. Then DAUF feeds the parameters, which capture key cues of CASSI degradation patterns and ill-posedness degree into each iteration to adaptively scale the linear projection and provide noise level information for the denoising network <ref:2205. p. 110102#pg3>. Secondly, we design a novel Half-Shuffle Transformer (HST) as the denoiser prior in each iteration <ref:2205. p. 110102#pg3>. Our HST can jointly extract local contextual information and model non-local dependencies while requiring much cheaper computational costs than global Transformer <ref:2205. p. 110102#pg3>. We achieve this by customizing a Half-Shuffle Multi-head Self-Attention (HS-MSA) mechanism that composes the basic unit of HST <ref:2205. p. 110102#pg6>. By plugging HST into DAUF, we establish an iterative architecture, Degradation-Aware Unfolding Half-Shuffle Transformer (DAUHST), for HSI reconstruction <ref:2205. p. 010102#pg3>.
--- Page 4 ---
In CASSI, we denote the vectorized measurement as y ∈ R n, where n = H(W +d(Nλ −1)) <ref:2205.10102#pg2>. Given the vectorized shifted HSI signal x ∈ R nNλ and the sensing matrix Φ ∈ R n×nNλ that is determined by the physical mask <ref:2205. p. 210102#pg3>. The degradation model of CASSI can be formulated as y = Φx + n, (1) where n ∈ R n represents the vectorized imaging noise on the measurement <ref:2205. p. 210102#pg3>. Then the task of HSI reconstruction is given y (captured by the camera) and Φ (calibrated based on pre-design), solving x <ref:2205. p. 210102#pg3>.
--- Page 5 ---
Previous unfolding frameworks [43-46] do not estimate the CASSI degradation patterns to adjust the iterative learning <ref:2205.10102#pg2>. To alleviate this limitation, we formulate a principled Degradation-Aware Unfolding Framework (DAUF) as depicted in Fig. p. 2 2 <ref:2205. p. 010102#pg2>. DAUF starts from MAP theory where the original HSI signal could be estimated by minimizing the following energy function as ˆx = arg min x 1/2 y − Φx2 + τR(x) <ref:2205. p. 210102#pg2>. This is reformulated using an auxiliary variable z, Eq. p. 2 (2) can be reformulated as ˆx = arg min x 1/2 y − Φx2 + τR(z), s. p. 2t. z = x <ref:2205. p. 010102#pg2>. To obtain an unfolding inference, we adopt half-quadratic splitting (HQS) algorithm for its simplicity and fast convergence <ref:2205. p. 210102#pg2>. Then Eq. (3) is solved by minimizing Lµ(x, z) = 1/2 y − Φx2 + τR(z) + µ 2 z − x2, where µ is a penalty parameter that forces x and z to approach the same fixed point <ref:2205. p. 210102#pg2>. This leads to iterative sub-problems (5), where k = 0, 1,. p. 2.., K - 1 indexes the iteration <ref:2205. p. 210102#pg2>. The data fidelity term is associated with a quadratic regularized least-squares problem, i. p. 2e., xk+1 in Eq. (5) has a closed-form solution <ref:2205. p. 210102#pg5>. By plugging ΦΦT into the closed-form solution, we obtain an efficient element-wise computation for xk+1 <ref:2205. p. 210102#pg5>. The penalty parameter µ should be large enough so that x and z can approach approximately the same fixed point, indicating that µ controls the convergence and output of each iteration <ref:2205. p. 310102#pg5>.
--- Page 6 ---
Returning to Eq. (5), we also set τ as iteration-specific parameters and zk+1 can be reformulated as zk+1 = arg min z 1/2(p τk+1/µk+1) 2 z − xk+12 + R(z) <ref:2205. p. 310102#pg5>. From the perspective of Bayesian probability, Eq. p. 3 (5) is equivalent to denoising image xk+1 with a Gaussian noise at level p τk+1/µk+1 [33] <ref:2205. p. 310102#pg5>. To conveniently solve Eq. p. 3 (5), we set 1 (√ τk+1/µk+1) 2 = µk+1/τk as parameters to be estimated from CASSI <ref:2205. p. 310102#pg5>. Let αk def = µk, βdef = µk/τk, and βdef = [β1,. p. 3,., βK] <ref:2205. p. 010102#pg5>. Then we can formulate our DAUF as an iterative scheme (α, β) = E(y, Φ), xk+1 = P(y, zk, αk+1, Φ), zk+1 = D(xk+1, βk+1) <ref:2205. p. 310102#pg5>.
--- Page 7 ---
Fig. 2 shows the architecture of E. p. 3 It consists of a conv1 × 1 a strided conv3 × 3 a global average pooling and three fully connected layers <ref:2205. p. 310102#pg8>. Through E, DAUF captures critical cues from CASSI by learning the degradation patterns and ill-posedness degree caused by the mask-modulation and dispersionintegration <ref:2205. p. 310102#pg8>. Parameters α and β estimated by E direct the iterative learning by adaptively scaling the linear projection in Eq. p. 3 (10) and providing noise level information for the denoising network <ref:2205. p. 110102#pg8>.
--- Page 8 ---
When designing the denoiser prior, previous deep unfolding methods mainly adopt CNNs, showing limitations in capturing long-range dependencies <ref:2205.10102#pg2>. To address these challenges, we propose Half-Shuffle Transformer (HST) to play the role of D network <ref:2205. p. 310102#pg2>. Our HST can jointly extract local contextual information and model non-local dependencies while requiring much cheaper computational costs than global Trans… p. 1
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Degradation-Aware Unfolding Framework (DAUF) allows for adaptive control of iterative learning by estimating
parameters from the compressed image and physical mask,
whichcapture key cues of CASSI degradation patterns and ill-posedness degree.
This enables the framework to adaptively scale the linear projection and provide noise level information for the denoising network in each iteration. -
Half-Shuffle Transformer (HST) can be integrated into DAUF to establish a novel Transformer-based deep unfolding method, Degradation-Aware Unfolding Half-Shuffle Transformer (DAUHST). HST
jointly extract local contextual information and model non-local dependencies,
utilizing aHalf Shuffle Multihead Self Attention (HS-MSA) mechanism that combines the advantages of global MSA [53] and local window-based MSA [51].
-
The DAUHST system can model dramatically outperform state-of-the-art (SOTA) deep unfolding methods while requiring
cheaper computational and memory costs.
Specifically, comparing DAUHST with CNN-based E2E methods shows it surpasses them by large margins while requiring less computation. -
The system can achieve superior reconstruction quality in real HSI applications, as shown in experiments where DAUHST is more robust to noise distortion and
more effective in real HSI reconstruction.
It canclearly reconstruct the picked flower at all wavelengths
in visual comparisons.
Sources
- GAP-net for Snapshot Compressive Imaging
- ViViT: A Video Vision Transformer
- XCiT: Cross-Covariance Image Transformers
- Swin-Unet: Unet-like Pure Transformer for Medical Image Segmentation
- TFPose: Direct Human Pose Estimation with Transformers
- Video Super-Resolution Transformer
- Flow-Guided Sparse Transformer for Video Deblurring
- Coarse-to-Fine Sparse Transformer for Hyperspectral Image Reconstruction
- RFormer: Transformer-based Generative Adversarial Network for Real Fundus Image Restoration on A New Clinical Benchmark
Related papers
- HDNet: High-resolution Dual-domain Learning for Spectral Compressive Imaging
- Mask-guided Spectral-wise Transformer for Efficient Hyperspectral Image Reconstruction
- FDIR: Fidelity-Anchored Decompressed Image Restoration for Human and Machine Perception
- A Disease-Centric Vision-Language Foundation Model for Precision Oncology in Kidney Cancer
- CADS: A Comprehensive Anatomical Dataset and Segmentation for Whole-Body Anatomy in Computed Tomography
- Task-Aware Bitrate Thresholds for Codec-Robust Ultrasound Segmentation