Structural Guidance for Unified Joint Demosaicing and Denoising
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Electrical Engineering. The day in electrical engineering: signals, systems and the hardware that carries them.
Rosa: I'm Rosa.
Dev: And I'm Dev, here with Taro.
Rosa: We're looking at the paper: "Structural Guidance for Unified Joint Demosaicing and Denoising".
Dev: Here's the gist of the paper. Joint demosaicing and denoising (JDD) remains challenging because different Bayer-like color filter arrays (CFAs) and sensor noise jointly corrupt both color sampling and image content <ref:2608.09995#pg2> Existing unified restoration networks are prone to structural degradation around edges, repetitive textures, and moiré patterns where local evidence is unreliable <ref:2608.09995#pg3> This paper proposes a structural-guided unified restoration framework that injects pretrained structural knowledge into CFA-aware image restoration <ref:2608.09995#pg4> Pruning redundant reasoning tokens The authors attribute the limitation of existing methods to the absence of explicit structural guidance beyond pixel-level reconstruction supervision <ref:2608.09995#pg5> They propose a structural-guided unified JDD framework that combines a pattern- and noiseconditioned restoration backbone with a frozen pretrained structural encoder that remains active during inference <ref:2608.09995#pg6> The model receives a unified five-channel observation consisting of the raw mosaic, CFA masks, and a noiselevel map <ref:2608.09995#pg7> A SwinIR restoration branch reconstructs pixel details under CFA-conditioned modulation <ref:2608.09995#pg8> A parallel structural reasoning branch extracts complementary structural cues from a sparse pseudo-RGB observation <ref:2608.09995#pg8> How the training signal is built The input representation consists of a unified five-channel tensor X =
Yein, Mvalp, Rmp, valGmp, valB, S: <ref:2608.09995#pg7> The sparse pseudo-RGB representation P places each observed measurement in its corresponding color channel while leaving the two unobserved channels at zero <ref:2608.09995#pg8> The restoration branch is redesigned as a pattern- and noise-aware backbone through heterogeneous sensor encoding, deep condition modulation, and native-grid feature preservation <ref:2608.09995#pg9> The conditioned Swin Transformer block uses a token-wise two-layer modulation MLP CM(F; C) = F ⊙ (1 + 0.1 tanh(γC)) + βC <ref:2608.09995#pg8> Where the gains come from The model achieves structural guidance by injecting a frozen DINOv3 encoder, instantiated with DINOv3 ViT-S/16, into the CFA-aware restoration without clean-target feature supervision while remaining active at inference <ref:2608.09995#pg9> The structural feature is integrated through restorationdependent residual fusion: ∆R = ffuse(
R, A: ), Rf = R + ∆ <ref:2608.09995#pg8> This online structural pathway converts pretrained patch tokens into restoration-compatible features and injects them as a learned residual correction <ref:2608.09995#pg9> The overall training objective is Ltotal = Lrgb + λknownLknown + λmosLmos <ref:2608.09995#pg8> Model-development analysis The model development analysis shows that adding the frozen structural encoder, shared trunk, and subsequent finetuning progressively raises the balanced PSNR from 32.461 to 32.525 dB <ref:2608.09995#pg8> The final configuration integrates these components and achieves the best balanced PSNR of 32.732 dB <ref:2608.09995#pg8> The full model result shows that the strongest performance is obtained when the complete architecture starts from an already reliable pixel-level JDD mapping <ref:2608.09995#pg8> Comparison cases win rate mean gain median gain The online Full model exceeds both listed noise-free-trained baselines in every paired case <ref:2608.09995#pg8> The aggregate MIT-Hard values show that Ours ranks first for every dataset–CFA combination in Tables 3 and 1 <ref:2608.09995#pg8> Our method outperforms ESUM by 3.84 dB and All-in-One SwinIR by 4.13 dB under synthetic sensor noise <ref:2608.09995#pg8> The positive lower endpoints of both gain ranges are important: even the least improved paired cases remain above the corresponding listed baseline <ref:2608.09995#pg8> The aggregate, group-wise, and paired results answer complementary parts of these questions and are interpreted only within their stated comparison protocols <ref:2608.09995#pg8> Limitations, Scope, and Ethics Current limitations include the use of synthetic Gaussian noise and sRGB-derived mosaics <ref:2608.09995#pg8> The architecture accepts alternative pretrained encoders through its adapter interface, although the experiments instantiate only DINOv3 <ref:2608.09995#pg8> Future work will evaluate other structural encoders, realistic signaldependent noise, and more efficient structural guidance <ref:2608.09995#pg8> The work uses public image-restoration benchmarks and involves no human participants or personal data collection <ref:2608.09995#pg8> Restored images should therefore not be treated as unqualified evidence in forensic, medical, or other high-stakes settings <ref:2608.09995#pg8> Additional visual comparisons Figure 8 complements the quantitative CFA analysis with fixed-crop comparisons across BSD100, Kodak24, and Urban100 and all three sampling families <ref:2608.09995#pg8> The selected crops emphasize thin boundaries, text-like strokes, and repetitive patterns that are diagnostically difficult for demosaicing <ref:2608.09995#pg8> These comparisons supplement the datasetlevel metrics by showing the spatial form of errors that a single PSNR or SSIM value cannot localize <ref:2608.09995#pg8> The primary reading criteria for both figures are edge continuity, preservation of thin strokes and repeated lines, suppression of zippering and false color, residual noise, and over-smoothing of fine texture <ref:2608.09995#pg8> Figure 11 reports the distribution of PSNR differences rather than only their overall mean <ref:2608.09995#pg8> The median gains are smaller than the means, indicating that several large improvements raise the average, but the 100% win rates show that those large cases are not the sole source of the aggregate advantage <ref:2608.09995#pg8> The case-level view therefore complements, rather than duplicates, the dataset-level averages <ref:2608.09995#pg8> Residual adaptation and shared decoding The structural fusion is residual by construction <ref:2608.09995#pg8> The shared decoder then exposes a common refined feature to both reconstruction objectives before the RGB and mosaic paths separate, allowing the auxiliary sensor-domain target to regularize shared content without forcing every RGB-specific decoder feature to serve the mosaic task <ref:2608.09995#pg8> This design interpretation is consistent with the monotonic development trend in Table 10 <ref:2608.09995#pg8> The final fusion convolution is zero-initialized, so ∆ = 0 and Rf = R at initialization <ref:2608.09995#pg8> The shared decoder trunk feeds an RGB reconstruction head and an auxiliary clean mosaic head, with the latter encouraging the fused representation to retain sensor-domain information <ref:2608.09995#pg8> Implementation distinction from DINOv2 To distinguish training-only feature supervision from the proposed online structural branch, we also consider a DINOv2 feature-distillation baseline <ref:2608.09995#pg8> The proposed online branch processes the noisy sparse observation, remains active at inference, and uses no feature-distillation objective <ref:2608.09995#pg8> The frozen structural encoder remains active during both training and inference, while the adapter, fusion module, and restoration network are jointly optimized using only the reconstruction objectives <ref:2608.09995#pg8> The reported ±49.08 ms is the population standard deviation over the 100 synchronized timed iterations <ref:2608.09995#pg8> The complexity values should accordingly be read as the end-toend cost of the proposed accuracy–structure trade-off, not as the cost of the restoration backbone alone <ref:2608.09995#pg8> Reproducibility guide Exact reproduction requires the conditioned SwinIR stream, DINOv3 adapter and residual fusion, shared decoder trunk, RGB decoder and head, and mosaic decoder and head <ref:2608.09995#pg8> The reproducible evaluation order is: acquire the public datasets and pretrained structural weights from their original sources; crop each image to the selected CFA period; construct the five-channel observation with the fixed RGGB phase, stated noise seed, and noise-map scaling <ref:2608.09995#pg8> This order keeps data synthesis, inference, metric conversion, and aggregation explicit <ref:2608.09995#pg8> The accompanying code archive provides a minimal executable demonstration rather than a redistribution of the full training repository <ref:2608.09995#pg8> Exact reproduction requires the conditioned SwinIR stream, DINOv3 adapter and residual fusion, shared decoder trunk, RGB decoder and head with no distillation head in the online model <ref:2608.09995#pg8> The frozen DINOv3 encoder is acquired separately from its original source and loaded before the remaining trained model state <ref:2608.09995#pg8> The accompanying README identifies the structural encoder source, pretrained weights, architecture flags, and evaluation configuration so that the intended online path can be instantiated without ambiguity <ref:2608.09995#pg8> The reproducible evaluation order is: acquire the public datasets and pretrained structural weights from their original sources; crop each image to the selected CFA period; construct the five-channel observation with the fixed RGGB phase, stated noise seed, and noise-map scaling <ref:2608.09995#pg8> This order keeps data synthesis, inference, metric conversion, and aggregation explicit which is necessary because changing any one of them can shift the reported values <ref:2608.09995#pg8> The accompanying code archive provides a minimal executable demonstration rather than a redistribution of the full training repository <ref:2608.09995#pg8> The implementation that accepts but silently ignores enable dinov3 branch or enable shared decoder trunk is incompatible with the reported architecture <ref:2608.09995#pg8> The frozen DINOv3 encoder is acquired separately from its original source and loaded before the remaining trained model state <ref:2608.
Paper summary: Rosa: So we're looking at this paper called "Structural Guidance for Unified Joint Demosaicing and Denoising." Basically, the problem they're tackling is that demosaicing—which is taking raw sensor data and turning it into a full color image—and denoising are both hard things to do together because the way the Bayer pattern is laid out, those color filter arrays, messes up both the color sampling and the actual picture content.
Dev: Right. So existing unified restoration networks try to model that Bayer pattern geometry directly, but they mostly train them using just pixel-level supervision. This means when things get tricky—like around edges or in areas with repetitive textures or moiré patterns—the local evidence is unreliable, and the network struggles because it hasn't learned the underlying structure of the image yet.
Taro: It seems like they’re saying that without some explicit structural guidance beyond just trying to reconstruct pixels correctly, these models are going to degrade those tricky areas. I mean, when you're testing this outside the lab, things get messy fast; you need something that understands the shape of the image even when the sensor data is noisy or ambiguous.
Rosa: Exactly. The authors attribute this limitation partly to not having any explicit structural guidance beyond that pixel-level reconstruction supervision which they do have in other methods one <ref:2608.09995#pg1,attribute this limitation partly to>. So what they propose is a new framework that injects some pretrained structural knowledge into this CFA-aware image restoration process one <ref:2608.09995#pg1>.
Dev: They design a unified five channel observation for their model. This input includes the raw mosaic, the Bayer pattern masks, and a map showing the noise level in the data one <ref:2608.09995#pg1>. This setup is what allows them to handle multiple color filter array layouts with a single network instead of needing separate models for every layout two <ref:2608.09995#pg3>.
Taro: When I look at that structure, they have a SwinIR branch handling the pixel details under CFA modulation, and then they have this parallel structural reasoning branch that pulls complementary structural cues from a sparse pseudo RGB observation one <ref:2608.09995#pg1>. That sounds like they're trying to look at the picture in two ways simultaneously.
Rosa: Right. So how do you build that signal for training? They use an input tensor X with five channels: the raw mosaic, the CFA masks, and a noise level map one <ref:2608.09995#pg1>. They then create a sparse pseudo RGB representation P where each measurement goes into its own color channel but leaves the other two channels at zero one <ref:2608.09995#pg1>.
Paper summary: Dev: That sparse pseudo RGB representation P is key because it lets them condition their restoration branch using that pattern information one <ref:2608.09995#pg1>. The restoration branch itself is redesigned to be pattern and noise aware through things like heterogeneous sensor encoding, deep condition modulation, and preserving native grid features three.
Taro: I see a token-wise two layer modulation MLP CM of the Swin Transformer block where it does this operation F times with the condition C: F multiplied by one plus zero point one times the tanh of gamma C plus beta C one <ref:2608.09995#pg1>. That sounds like they're modulating the features based on both the pattern and noise information.
Rosa: And this is where they get their structural guidance. They inject a frozen DINOv3 encoder, which is instantiated with DINOv3 ViT S over sixteen into the CFA-aware restoration process even though there isn't clean target feature supervision one. This structural encoder stays active during inference, which is important for deployment nine.
Dev: They integrate this structural feature through something called restoration dependent residual fusion. They define a residual Delta R by fusing the pattern and noise aware restoration details, then they calculate an updated R by adding that delta to the original R eight. This online structural pathway takes those pretrained patch tokens and turns them into features compatible with reconstruction, injecting them as a learned residual correction nine.
Taro: So this frozen encoder isn't just sitting there; it’s generating these structural features that get fused in during the actual restoration process. It sounds like they are using the structural knowledge to guide the reconstruction of details around those hard areas we talked about earlier ten.
Rosa: And their overall training objective is a combination: L total equals L rgb plus lambda known L known plus lambda mos L mos eight. They're balancing the standard RGB reconstruction loss with some other losses related to known data and mosaic noise levels eight.
Dev: Looking at the development analysis, adding that frozen structural encoder, a shared trunk, and then fine-tuning it progressively raises the balanced PSNR from thirty-two point four six one dB to thirty-two point five two five dB eight. The final configuration they showed achieves a best balanced PSNR of thirty-two point seven three two dB eight.
Paper summary: Taro: So, when I look at the comparison cases, their online full model outperforms both noise free trained baselines in every paired case eight. Even when you look at the aggregate MIT hard values across all datasets and CFA combinations, theirs ranks first in tables three and one eight.
Rosa: They also show they outperform ESUM by three point eight four dB and All-in-One SwinIR by four point one three dB under synthetic sensor noise eight. It’s not just about the average gains; even the least improved paired cases are still above the baseline values, which shows that their method is consistent across different scenarios eight.
Dev: The paper also looks at visual comparisons, figure eight shows fixed crop comparisons across BSD100, Kodak24 and Urban100 for all three sampling families eight. Those crops focus on thin boundaries and text like strokes which are diagnostically hard to demosaic because a simple PSNR score won't show you the spatial form of those errors eight.
Taro: The primary reading criteria they use there are edge continuity, preserving those thin strokes and repeated lines, suppressing zippering and false color, residual noise, and over-smoothing fine texture eight. That’s the kind of detail that matters for real image quality.
Rosa: They also show the distribution of PSNR differences in figure eleven instead of just a mean value eight. The median gains are smaller than the means, which suggests that some very large improvements raise the average, but those big cases aren't the only source of their overall advantage eight.
Dev: This leads to thinking about what this actually means for us. They’re using a frozen DINOv3 encoder without clean target feature supervision but still getting strong results because they use that structural knowledge online one <ref:2608.09995#pg1>. It’s an interesting way to get structure without needing perfect ground truth features during training eight.
Taro: The paper mentions their limitation, which is using synthetic Gaussian noise and sRGB derived mosaics for testing eight. They also state that the architecture accepts alternative pretrained encoders through its adapter interface, but in this study they only used DINOv3 eight.
Rosa: And it’s clear what they don't cover. Future work will need to evaluate other structural encoders, more realistic signal dependent noise, and find ways to make this structural guidance even more efficient eight. They also stress that because they use public benchmarks with no human participants or personal data collection, restored images shouldn't be treated as unqualified evidence in forensic or medical settings eight.
Paper summary: Dev: So to wrap up the whole thing, the paper "Structural Guidance for Unified Joint Demosaicing and Denoising" proposes this structural-guided unified restoration framework that injects pretrained structural knowledge into CFA aware image restoration one <ref:2608.09995#pg1,structural-guided unified restoration framework that injects pretrained structural knowledge into CFA>. It does this by combining a pattern and noise conditioned backbone with a frozen pretrained structural encoder that stays active during inference one <ref:2608.09995#pg1>.
Taro: And the core idea is using residual fusion to convert those frozen patch tokens into features for correction, which is then used alongside the RGB reconstruction head to regularize shared content eight. It’s about making sure the reconstruction respects the underlying visual structure of the image, not just local pixel errors.
Rosa: And what this means simply for us is that we can get better results in joint demosaicing and denoising by not just focusing on fixing pixel-level noise but by giving the model some prior knowledge about how images are actually supposed to look structurally one <ref:2608.09995#pg1>. It’s a move from purely data-driven learning to incorporating meaningful visual structure into the restoration loop.
Dev: The authors of "Structural Guidance for Unified Joint Demosaicing and Denoising" have essentially shown that injecting pretrained structural knowledge, specifically from a model like DINOv3, into a CFA aware restoration network can improve performance by providing better guidance on edge continuity and texture preservation one <ref:2608.09995#pg1,Structural Guidance for Unified Joint Demosaicing and Denoising>.
Taro: For the listener listening to this, the implication is that when you’re dealing with tricky sensor noise or complex color sampling patterns, having an extra layer of structural understanding helps stabilize the reconstruction of fine details where standard methods fail ten.
Rosa: So, in conclusion, this paper suggests a unified framework that handles different color filter arrays while using external structural knowledge to guide the restoration process rather than relying solely on local pixel evidence one <ref:2608.09995#pg1>. It’s a way to handle the complexity of demosaicing and denoising together more robustly.
Dev: That's the gist of "Structural Guidance for Unified Joint Demosaicing and Denoising." They moved beyond just learning how to fix pixels to learning how those pixels should be organized structurally, which gives a clearer path forward for these kinds of image processing tasks one <ref:2608.09995#pg1>.
Conclusion: Rosa: So we're wrapping up this look at "Structural Guidance for Unified Joint Demosaicing and Denoising." Basically, this paper is about fixing the problem where demosaicing and denoising get messy when you have different camera sensor layouts mixed with noise in the data.
Dev: Yeah, it focuses on making a unified restoration framework that handles those different color filter arrays without needing separate models for every single one. It’s trying to bridge that gap between color sampling and image content corruption.
Rosa: Right. The authors are injecting some pretrained structural knowledge into this process, using something like DINOv3, which is a powerful vision model we already have trained on massive amounts of images.
Dev: And they do it by creating a new way to fuse information during the restoration—a residual fusion—that takes those structural cues and uses them to correct the pixel errors in real-time.
Rosa: It’s like giving the image a built-in blueprint so it knows how things should look structurally, even when the raw sensor data is a bit noisy or ambiguous.
Dev: But they also have to worry about how this runs in practice. They show that adding this structural guidance improves performance on standard metrics, but they also point out that their testing only used synthetic noise and certain sRGB mosaics.
Rosa: That’s true. The authors themselves mention that because they didn't use real-world signal dependent noise or human data, we shouldn't treat the restored images as perfect evidence in high-stakes fields like medicine or forensics.
Dev: Exactly. The paper shows a strong performance improvement, but you have to keep an eye on those limitations when you think about deploying this kind of system for actual use outside of a controlled lab setting.
Rosa: It’s interesting because it shows how we can combine different types of learning—the pattern recognition from the restoration branch and the structural understanding from that frozen encoder—to get a better picture overall.
Dev: So, the takeaway here is that incorporating prior knowledge about image structure helps stabilize joint demosaicing and denoising when dealing with complex sensor data. Next up, we’re going to look at how this actually performs compared to other methods on real datasets.
Harbin Institute of Technology (Shenzhen)
eess.IV, cs.CV
Submitted: 2026-08-07
Updated: 2026-11-11
Comments: 19 pages, including supplementary material
Code: https://github.com/Calista-pedi/SG-for-JDD
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 91/100
The gist: Joint demosaicing and denoising (JDD) remains challenging because different Bayer-like color filter arrays (CFAs) and sensor noise jointly corrupt both color sampling and image content…
Key concepts
- Joint Demosaicing and Denoising (JDD)
- This is the task of simultaneously reconstructing color information (demosaicing) and removing noise from an image. It is challenging because different sensor patterns (color filter arrays) and noise independently damage both the color sampling accuracy and the image content itself, requiring a unified approach.
- Structural-Guided Unified Restoration Framework
- This framework combines a pattern- and noise-conditioned restoration backbone with a frozen pretrained structural encoder. The encoder provides learned structural knowledge that is injected into the restoration process, helping to correct errors like edge degradation and moiré patterns where local evidence is unreliable.
- Online Structural Pathway
- This mechanism converts the pretrained patch tokens from the structural encoder into features compatible with image restoration. These features are then injected as a learned residual correction ($ riangle R = ffuse([R, A])$), allowing the model to adapt its reconstruction based on external structural cues during inference.
Terminology
Summary
Joint demosaicing and denoising (JDD) remains challenging because different Bayer-like color filter arrays (CFAs) and sensor noise jointly corrupt both color sampling and image content <ref:2608. p. 0 zeng2018@gmail. com
Abstract
Joint demosaicing and denoising is a fundamental step in
camera image signal processing, yet remains challenging because different Bayer-like color filter arrays (CFAs) and sensor
noise jointly corrupt both color sampling and image content. com
Abstract
Joint demosaicing and denoising is a fundamental step in
camera image signal processing, yet remains challenging because different Bayer-like color filter arrays (CFAs) and sensor
noise jointly corrupt both color sampling and image content. Existing unified restoration networks explicitly model CFA
geometry but are still driven primarily by pixel-level supervision, making them prone to structural degradation around
edges, repetitive textures, and moiré patterns where local evidence is unreliable. We attribute this limitation partly to the
absence of explicit structural guidance beyond pixel-level reconstruction supervision. Motivated by this observation, we
propose a structural-guided unified restoration framework that
injects pretrained structural knowledge into CFA-aware image
restoration.Where the paper says it
Where the paper says it
Where the paper says it
Pruning redundant reasoning tokens
The authors attribute the limitation of existing methods to the absence of explicit structural guidance beyond pixel-level reconstruction supervision <ref:2608.09995#pg5> They propose a structural-guided unified JDD framework that combines a pattern- and noiseconditioned restoration backbone with a frozen pretrained structural encoder that remains active during inference <ref:2608. p. 1 Unified JDD must therefore address not only
the variation across CFA layouts, but also the structural ambiguity that remains after the sampling configuration is known,
as illustrated in Fig. To address this limitation, we propose a structurally guided
unified JDD framework that combines a pattern- and noiseconditioned restoration backbone with a frozen pretrained
structural encoder that remains active during inference. Motivated by this observation, we
propose a structural-guided unified restoration framework that
injects pretrained structural knowledge into CFA-aware image
restoration. Our model receives a unified five-channel observation consisting of the raw mosaic, CFA masks, and a noiselevel map. Our model receives a unified five-channel observation consisting of the raw mosaic, CFA masks, and a noiselevel map. A SwinIR restoration branch reconstructs pixel
details under CFA-conditioned modulation, while a parallel
structural reasoning branch extracts complementary structural cues from a sparse pseudo-RGB observation. Our model receives a unified five-channel observation consisting of the raw mosaic, CFA masks, and a noiselevel map. A SwinIR restoration branch reconstructs pixel
details under CFA-conditioned modulation, while a parallel
structural reasoning branch extracts complementary structural cues from a sparse pseudo-RGB observation.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
How the training signal is built
The input representation consists of a unified five-channel tensor X = [Yein, Mvalp, Rmp, valGmp, valB, S] <ref:2608.09995#pg7> The sparse pseudo-RGB representation P places each observed measurement in its corresponding color channel while leaving the two unobserved channels at zero <ref:2608. p. 2 The five-channel tensor X ∈ R
H×W×5
contains the
noisy mosaic, three CFA masks, and the noise map, and is
used by the restoration branch. The sparse pseudo-RGB representation P ∈ R
H×W×3 places each observed measurement in its corresponding color channel while leaving the
two unobserved channels at zero. The main contributions are summarized as follows:
• We identify insufficient explicit structural guidance beyond pixel-level supervision as an important limitation
of unified JDD, and introduce a structural branch that injects pretrained dense representations—instantiated with
a frozen DINOv3 encoder—into CFA-aware restoration
without clean-target feature supervision while remaining
active at inference. • We redesign SwinIR as a pattern- and noise-aware unified JDD backbone through heterogeneous sensor encoding, deep condition modulation, and native-grid feature
preservation, enabling one model to adapt to different
CFA families, spatial phases, and noise levels. 2023; Tedla et al. Pattern-specific
networks have improved reconstruction for individual QuadarXiv:2608.Where the paper says it
Where the paper says it
Where the paper says it
Where the gains come from
The model achieves structural guidance by injecting a frozen DINOv3 encoder, instantiated with DINOv3 ViT-S/16, into the CFA-aware restoration without clean-target feature supervision while remaining active at inference <ref:2608.09995#pg9> The structural feature is integrated through restorationdependent residual fusion: ∆R = ffuse([R, A]), Rf = R + ∆ <ref:2608. p. 3 This produces an adapted
structural feature A ∈ R
H×W×96
. The structural feature is integrated through restorationdependent residual fusion:
∆R = ffuse([R, A]), Rf = R + ∆R, (4)
where [·, ·] denotes channel-wise concatenation. Directly transferring pretrained visual features is challenging because the sparse
pseudo-RGB observation P differs from the dense natural
images used for pretraining, while the resulting patch representation is not aligned with R in spatial resolution, channel
width, or feature statistics. We therefore introduce an online
structural pathway that converts pretrained patch tokens into
restoration-compatible features and injects them as a learned
residual correction. The
full-image loss Lrgb supervises the restored RGB output,
Lknown places additional emphasis on retained sensor-color
locations, and Lmos supervises the auxiliary clean mosaic
prediction. The overall training objective is
L = Lrgb + λknownLknown + λmosLmos, (5)
where λknown and λmos control the retained-measurement
and auxiliary mosaic losses, respectively.Where the paper says it
Where the paper says it
Where the paper says it
Model-development analysis
The model development analysis shows that adding the frozen structural encoder, shared trunk, and subsequent finetuning progressively raises the balanced PSNR from 32.461 to 32.525 dB <ref:2608. p. 0 2023; Tedla et al. Pattern-specific
networks have improved reconstruction for individual QuadarXiv:2608. Adding the
frozen structural encoder, shared trunk, and subsequent finetuning progressively raises the balanced PSNR from 32. The final configuration integrates these components and achieves the best balanced PSNR of 32. 2023; Tedla et al. Pattern-specific
networks have improved reconstruction for individual QuadarXiv:2608. Sequential
variants trace the progression from the dual-head baseline to
structural guidance, shared decoding, and fine-tuning, while
the Full model achieves the best performance. The full-model result shows that the strongest performance is obtained when the complete architecture starts
from an already reliable pixel-level JDD mapping.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Comparison cases win rate mean gain median gain
The online Full model exceeds both listed noise-free-trained baselines in every paired case <ref:2608.09995#pg8> The aggregate MIT-Hard values show that Ours ranks first for every dataset–CFA combination in Tables 3 and 1 <ref:2608. p. 4 Quantitative and Qualitative Results
Zero-Noise Results. Ours ranks first for every dataset–
CFA combination in Tables 3 and 1. 4 and 6 evaluate robustness under synthetic sensor noise. Averaged over three
CFAs and three nonzero noise levels, Our method outperforms ESUM by 3. The main contributions are summarized as follows:
• We identify insufficient explicit structural guidance beyond pixel-level supervision as an important limitation
of unified JDD, and introduce a structural branch that injects pretrained dense representations—instantiated with
a frozen DINOv3 encoder—into CFA-aware restoration
without clean-target feature supervision while remaining
active at inference. • We redesign SwinIR as a pattern- and noise-aware unified JDD backbone through heterogeneous sensor encoding, deep condition modulation, and native-grid feature
preservation, enabling one model to adapt to different
CFA families, spatial phases, and noise levels. Table 2 and Figs. 4 and 6 evaluate robustness under synthetic sensor noise. The experimental evidence is organized around four
questions: whether the unified architecture behaves consistently across Single-, Quad-, and Nona-Bayer observations;
whether performance degrades predictably as noise and CFA
block size increase; whether the reported validation gains
persist across individual cases rather than only in aggregate
averages; and what computation is required by the online
structural branch. Aggregate, group-wise, and paired results
answer complementary parts of these questions and are interpreted only within their stated comparison protocols.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Limitations, Scope, and Ethics
Current limitations include the use of synthetic Gaussian noise and sRGB-derived mosaics <ref:2608.09995#pg8> The architecture accepts alternative pretrained encoders through its adapter interface, although the experiments instantiate only DINOv3 <ref:2608. p. 6 Current limitations include the use of
synthetic Gaussian noise and sRGB-derived mosaics, the absence of calibrated real multi-CFA RAW data, and the inference cost of the structural pathway. The architecture accepts
alternative pretrained encoders through its adapter interface,
although the experiments instantiate only DINOv3. The architecture accepts
alternative pretrained encoders through its adapter interface,
although the experiments instantiate only DINOv3. Future
work will evaluate other structural encoders, realistic signaldependent noise, and more efficient structural guidance.Where the paper says it
Where the paper says it
Additional visual comparisons
Figure 8 complements the quantitative CFA analysis with fixed-crop comparisons across BSD100, Kodak24, and Urban100 and all three sampling families <ref:2608.09995#pg8> The selected crops emphasize thin boundaries, text-like strokes, and repetitive patterns that are diagnostically difficult for demosaicing <ref:2608. p. 0 (2020), and the joint
demosaicing, denoising, and super-resolution framework of
Xing and Egiazarian (2021). Despite this progress, thin or
oblique edges, repetitive textures, and moiré-prone patterns
remain difficult to reconstruct, often resulting in false colors,
zippering artifacts, distorted periodic structures, and oversmoothed details (Gharbi et al. Protocol and Metrics. We report RGB PSNR and
SSIM (Wang et al. Figure 7 complements the dataset averages with difficult MIT-Hard crops. The baselines often
introduce false-color moiré, zippering, or distorted periodic
patterns, whereas our method better preserves line spacing,
edge continuity, and color organization. The full-model result shows that the strongest performance is obtained when the complete architecture starts
from an already reliable pixel-level JDD mapping. We therefore attribute the large difference from external baselines to
the complete system and training recipe, while treating DINOv3 as a complementary structural component rather than
the sole source of the gain. The full-model result shows that the strongest performance is obtained when the complete architecture starts
from an already reliable pixel-level JDD mapping. We therefore attribute the large difference from external baselines to
the complete system and training recipe, while treating DINOv3 as a complementary structural component rather than
the sole source of the gain.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Residual adaptation and shared decoding
The structural fusion is residual by construction <ref:2608.09995#pg8> The shared decoder then exposes a common refined feature to both reconstruction objectives before the RGB and mosaic paths separate, allowing the auxiliary sensor-domain target to regularize shared content without forcing every RGB-specific decoder feature to serve the mosaic task <ref:2608. p. 1 A lightweight convolutional
adapter maps the pretrained patch representation to the spatial resolution and channel space of the restoration feature,
after which a restoration-dependent residual fusion module
injects the adapted representation as a structural correction
rather than replacing the sensor-aligned pathway. A shared
decoder trunk feeds an RGB reconstruction head and an auxiliary mosaic head, with the latter encouraging the fused representation to retain sensor-domain information. Scenes 1 and 2 respectively use Quad- and Nona-Bayer
observations. Developmental and Matched-Start Analysis
Table 4 traces the development from the dual-head baseline to the complete structurally guided model. The pseudo-RGB observation P is resized to 224 × 224 and normalized using
the fixed pretrained input statistics. The final patch tokens
are reshaped into a 14 × 14 spatial grid, bilinearly resized
to H × W, and projected from 384 to 96 channels by a
lightweight adapter consisting of a 1×1 convolution, GELU
activation, and a 3×3 convolution. A lightweight convolutional
adapter maps the pretrained patch representation to the spatial resolution and channel space of the restoration feature,
after which a restoration-dependent residual fusion module
injects the adapted representation as a structural correction
rather than replacing the sensor-aligned pathway. A shared
decoder trunk feeds an RGB reconstruction head and an auxiliary mosaic head, with the latter encouraging the fused representation to retain sensor-domain information.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Implementation distinction from DINOv2
To distinguish training-only feature supervision from the proposed online structural branch, we also consider a DINOv2 feature-distillation baseline <ref:2608.09995#pg8> The proposed online branch processes the noisy sparse observation, remains active at inference, and uses no feature-distillation objective <ref:2608. p. 3 Online Structural Prior Adaptation and Fusion
Let R ∈ R
H×W×96 denote the sensor-aligned feature produced by the restoration branch. Directly transferring pretrained visual features is challenging because the sparse
pseudo-RGB observation P differs from the dense natural
images used for pretraining, while the resulting patch representation is not aligned with R in spatial resolution, channel
width, or feature statistics. Conditioning ∆R jointly on R and A
allows the model to incorporate pretrained structural context
without directly replacing the restoration representation. The frozen structural encoder remains active during both
training and inference, while the adapter, fusion module, and Early dual-head, DINOv3-branch, and shared-decoder
variants were initialized sequentially from the earlier checkpoint, whereas the final model was rebuilt from SwinIRJDD. The early sequence is therefore reported as a developmental analysis, not as a strict one-factor additive ablation. We denote the output of the modified
restoration branch by
R = Hres
Yein, P, Mval
p
, S
∈ R
H×W×C , (3)
where C is the restoration feature width. Together, the multi-source encoding, deep sensorcondition modulation, and native-grid feature preservation
transform SwinIR from a conventional dense-image restorer
Figure 4: PSNR comparison for joint demosaicing and denoising under Gaussian noise with standard deviation σ.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Reproducibility guide
Exact reproduction requires the conditioned SwinIR stream, DINOv3 adapter and residual fusion, shared decoder trunk, RGB decoder and head, and mosaic decoder and head <ref:2608.09995#pg8> The reproducible evaluation order is: acquire the public datasets and pretrained structural weights from their original sources; crop each image to the selected CFA period; construct the five-channel observation with the fixed RGGB phase, stated noise seed, and noise-map scaling <ref:2608. p. 2 From the same sensor observation, we construct two input
representations:
X = [Yein, Mval
p
, S],
P = Mval
p ⊙ Rep3
Yein
,
(1)
where S is a normalized noise-level map and Rep3
(·) replicates a single-channel image along the RGB channel dimension. The five-channel tensor X ∈ R
H×W×5
contains the
noisy mosaic, three CFA masks, and the noise map, and is
used by the restoration branch. From the same sensor observation, we construct two input
representations:
X = [Yein, Mval
p
, S],
P = Mval
p ⊙ Rep3
Yein
,
(1)
where S is a normalized noise-level map and Rep3
(·) replicates a single-channel image along the RGB channel dimension. The five-channel tensor X ∈ R
H×W×5
contains the
noisy mosaic, three CFA masks, and the noise map, and is
used by the restoration branch. Section specifies the observation synthesis, conditioned restoration stream, online
structural reasoning branch, shared dual-head decoder, and
training objectives. Section documents the datasets, staged
optimization, model selection, metric implementation, and
comparison scope. The full-model result shows that the strongest performance is obtained when the complete architecture starts
from an already reliable pixel-level JDD mapping. We therefore attribute the large difference from external baselines to
the complete system and training recipe, while treating DINOv3 as a complementary structural component rather than
the sole source of the gain. The full-model result shows that the strongest performance is obtained when the complete architecture starts
from an already reliable pixel-level JDD mapping. We therefore attribute the large difference from external baselines to
the complete system and training recipe, while treating DINOv3 as a complementary structural component rather than
the sole source of the gain. A lightweight convolutional
adapter maps the pretrained patch representation to the spatial resolution and channel space of the restoration feature,
after which a restoration-dependent residual fusion module
injects the adapted representation as a structural correction
rather than replacing the sensor-aligned pathway. A shared
decoder trunk feeds an RGB reconstruction head and an auxiliary mosaic head, with the latter encouraging the fused representation to retain sensor-domain information. The main contributions are summarized as follows:
• We identify insufficient explicit structural guidance beyond pixel-level supervision as an important limitation
of unified JDD, and introduce a structural branch that injects pretrained dense representations—instantiated with
a frozen DINOv3 encoder—into CFA-aware restoration
without clean-target feature supervision while remaining
active at inference. • We redesign SwinIR as a pattern- and noise-aware unified JDD backbone through heterogeneous sensor encoding, deep condition modulation, and native-grid feature
preservation, enabling one model to adapt to different
CFA families, spatial phases, and noise levels. The main contributions are summarized as follows:
• We identify insufficient explicit structural guidance beyond pixel-level supervision as an important limitation
of unified JDD, and introduce a structural branch that injects pretrained dense representations—instantiated with
a frozen DINOv3 encoder—into CFA-aware restoration
without clean-target feature supervision while remaining
active at inference. • We redesign SwinIR as a pattern- and noise-aware unified JDD backbone through heterogeneous sensor encoding, deep condition modulation, and native-grid feature
preservation, enabling one model to adapt to different
CFA families, spatial phases, and noise levels. Unified JDD must therefore address not only
the variation across CFA layouts, but also the structural ambiguity that remains after the sampling configuration is known,
as illustrated in Fig. To address this limitation, we propose a structurally guided
unified JDD framework that combines a pattern- and noiseconditioned restoration backbone with a frozen pretrained
structural encoder that remains active during inference. Early dual-head, DINOv3-branch, and shared-decoder
variants were initialized sequentially from the earlier checkpoint, whereas the final model was rebuilt from SwinIRJDD. The early sequence is therefore reported as a developmental analysis, not as a strict one-factor additive ablation.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Structural Guidance Injection into Restoration: The system can inject pretrained structural knowledge by injecting a frozen DINOv3 encoder's dense structural representations into CFA-aware restoration without clean-target feature supervision while remaining active at inference.
-
Unified Backbone Redesign: The SwinIR backbone is redesigned as a pattern- and noise-aware unified JDD backbone through heterogeneous sensor encoding, deep condition modulation, and native-grid feature preservation, enabling one model to adapt to different CFA families, spatial phases, and noise levels.
-
Online Structural Prior Adaptation: The system can adapt pretrained patch tokens from a sparse pseudo-RGB projection into restoration-compatible features and inject them as a learned residual correction via the formula Delta R = ffuse([R, A]), where [·, ·] denotes channel-wise concatenation.
-
Enhanced Robustness to Ambiguity: The improved system can better resolve ambiguous fine patterns than previous methods by using structural guidance to provide complementary contextual cues for latent edges and repetitive structures when local CFA measurements are unreliable.
-
Noise Robustness Improvement: The model can exhibit consistent improvements over state-of-the-art unified and CFA-specific methods across multiple CFA patterns and noise levels, demonstrating enhanced robustness under synthetic Gaussian noise.
Abstract
Joint demosaicing and denoising is a fundamental step in camera image signal processing, yet remains challenging because different Bayer-like color filter arrays (CFAs) and sensor noise jointly corrupt both color sampling and image content. Existing unified restoration networks explicitly model CFA geometry but are still driven primarily by pixel-level supervision, making them prone to structural degradation around edges, repetitive textures, and moiré patterns where local evidence is unreliable. We attribute this limitation partly to the absence of explicit structural guidance beyond pixel-level reconstruction supervision. Motivated by this observation, we propose a structural-guided unified restoration framework that injects pretrained structural knowledge into CFA-aware image restoration. Our model receives a unified five-channel observation consisting of the raw mosaic, CFA masks, and a noise-level map. A SwinIR restoration branch reconstructs pixel details under CFA-conditioned modulation, while a parallel structural reasoning branch extracts complementary structural cues from a sparse pseudo-RGB observation. To bridge the substantial domain gap between sparse noisy sensor data and the natural-image pretraining domain of the structural encoder, we introduce a lightweight trainable adapter before residually fusing structural and restoration features. A shared decoder jointly predicts the restored RGB image and an auxiliary clean mosaic, providing supervision in both image and sensor domains. Extensive experiments across multiple CFA patterns and noise levels demonstrate consistent improvements over state-of-the-art unified and CFA-specific methods, indicating that adapted structural priors can enhance robust camera image restoration. The source codes and dataset are provided in the supplementary material.
Sources
Related papers
- HDNet: High-resolution Dual-domain Learning for Spectral Compressive Imaging
- Mask-guided Spectral-wise Transformer for Efficient Hyperspectral Image Reconstruction
- FDIR: Fidelity-Anchored Decompressed Image Restoration for Human and Machine Perception
- A Disease-Centric Vision-Language Foundation Model for Precision Oncology in Kidney Cancer
- CADS: A Comprehensive Anatomical Dataset and Segmentation for Whole-Body Anatomy in Computed Tomography
- Task-Aware Bitrate Thresholds for Codec-Robust Ultrasound Segmentation