Combining Pre- and Post-Demosaicking Noise Removal for RAW Video

arXiv:2410.02572 · eess.IV, cs.CV · Submitted 2024-10-03 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Electrical Engineering. The day in electrical engineering: signals, systems and the hardware that carries them.

Rosa: We're looking at the paper: "Combining Pre- and Post-Demosaicking Noise Removal for RAW Video".

Paper summary: Rosa: So we've just talked about the core thesis of this paper on combining pre and post demosaicking denoising for RAW video. Essentially, they show that mixing both types of denoisers works better than sticking to just one approach, because the optimal balance changes depending on how much noise is present in the scene.

Dev: And they found an empirical finding that higher noise levels actually benefit from a stronger influence of the pre-demosaicking stage. That means if you've got a lot of noise, you should lean into that initial filtering more heavily.

Taro: They also integrated these temporal trajectory prefiltering steps before each denoiser, which they claim further improves how texture is reconstructed in the final image.

Rosa: It sounds like the paper suggests that this combined self-similarity based scheme only needs an estimation of the noise model at the sensor and can adapt to any noise level, making it competitive with what's out there now.

Dev: And they compared their method against state-of-the-art deep learning methods, finding it outperforms FlexISP, RViDeNet and UDVD in all metrics.

Taro: They also matched or surpassed MaskDnGAN in PSNR for ISOs between three thousand two hundred and twelve thousand eight hundred and SSIM and MS-SSIM for the range of one thousand six hundred to six thousand four hundred while staying close to it in VMAF.

Rosa: So what they're really highlighting is that this approach offers a good trade-off when you need robust denoising across different kinds of sequences and noise levels that current neural networks still struggle with.

Dev: It really puts the focus on making classical video denoising algorithms viable alternatives for general video tasks, especially considering the computational constraints of mobile devices mentioned in their work.

Conclusion: Rosa: To wrap up, this paper about combining pre and post demosaicking denoising for RAW Video by Sánchez-Beeckman et al. shows that a balanced approach is superior to using just one stage of denoising before or after demosaicking.

Dev: The key finding they highlight is that the optimal balance depends on the noise level, specifically suggesting that you should increase the RAW filtering before demosaicking when you're dealing with higher ISO values.

Taro: They also emphasized how integrating those temporal trajectory prefiltering steps helps a lot with recovering fine details, especially when motion models are well-restricted to keep things aligned.

Rosa: And what this means for us is that this method only requires estimating the noise model at the sensor, which can be done with a simple camera calibration procedure, allowing accurate noise removal regardless of how high the noise level gets.

Dev: It's a practical point because it makes it suitable for real-world videography, meaning you don't need massive datasets to train something that works across all conditions.

Taro: The paper also flags a limitation: they found that with strong restrictions on the motion model, this temporal prefiltering step is key for detail recovery, but if those motion models aren't tight enough, the misalignment could cause issues.

Rosa: That's the trade-off they identified; you get better detail recovery by adding that temporal step, but you have to be careful about how strictly you define the motion model.

Dev: So in a nutshell, this work is about giving us a robust way to clean RAW video data by intelligently combining different denoising techniques while keeping the complexity manageable for practical use.

Institute of Applied Computing and Community Code (IAC3) · University of les Illes Balears · Huawei Technologies France

eess.IV, cs.CV

Submitted: 2024-10-03

Updated: 2024-11-29

Comments: 16 pages, 9 figures

DOI: 10.1109/TIP.2025.3527886

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 79/100

The gist: Combining Pre- and Post-Demosaicking Noise Removal for RAW Video is a self-similaritybased denoising scheme that weights both a pre- and a postdemosaicking denoiser for Bayer-patterned CFA video data…

Key concepts

Pre- and Post-Demosaicking Denoising
This is a two-stage denoising process applied to RAW video data. The first stage denoises the data while it is still mosaicked (before demosaicking), and the second stage denoises the demosaicked image. Finding the right balance between these two stages significantly improves final image quality.
Noise Model Estimation
The paper estimates sensor noise using a normal approximation of a Poisson distribution. This requires capturing flat-field and bias frames to fit parameters (a and b) for different ISO settings. A local variance estimator is then computed during denoising to accurately model the noise in any region.
Temporal Trajectory Prefiltering
Before each denoising stage, the method uses motion trajectories from surrounding frames to prefilter the data. This involves calculating optical flow and spatially aligning frames. This step helps reconstruct fine details by exploiting temporal redundancy, particularly at high noise levels.

Terminology

Summary

Combining Pre- and Post-Demosaicking Noise Removal for RAW Video is a self-similaritybased denoising scheme that weights both a pre- and a postdemosaicking denoiser for Bayer-patterned CFA video data We show that a balance between the two leads to better image quality, and we empirically find that higher noise levels benefit from a higher influence pre-demosaicking We also integrate temporal trajectory prefiltering steps before each denoiser, which further improve texture reconstruction The proposed method only requires an estimation of the noise model at the sensor, accurately adapts to any noise level, and is competitive with the state of the art, making it suitable for real-world videography

Methodology Overview

The proposed pipeline denoises each frame It ∈ I sequentially using spatio-temporal information from its surrounding frames by employing a modular approach with two denoising blocks at different stages of the ISP We denoise the frames both when they are still mosaicked and after demosaicking, gradually reducing the amount of noise while preserving the details of the scene The overall architecture involves spatially subsampling frames and packing them into 4-channel images according to their CFA structure, followed by a transformation TYUVW We propose a combined twostage denoising procedure where after a first denoising stage on the packed CFA data in the YUVW domain (Denoising 1 in Fig. 1), we return a fraction α of the noise This way, we restore details that may have been oversmoothed, and we facilitate a more precise second denoising stage (Denoising 2 in Fig. p. 3

Where the paper says it

1), we return a fraction α of the noise—defined as the difference between the noisy and the denoised image. That way, we restore details that may have been oversmoothed, and we facilitate a more precise second denoising stage (Denoising 2 in Fig.

1) on the now less noisy demosaicked frames

Noise Modeling and Preprocessing

The sensor noise model is described by the normal approximation of the Poisson distribution, where a read pixel xread is thus usually described as xread ∼ N (xtrue + O, axtrue + b) The noise parameters a and b for a particular ISO can be estimated by capturing flat-field frames and bias frames in specialized settings and fitting the observations to a linear model We compute a local variance estimator for each patch neighbourhood while denoising it, which is given by equation (2) The noise distribution in the transformed domain follows from equation (3)

Temporal Information Exploitation

To exploit temporal redundancy, we establish motion trajectories of patches over a temporal neighbourhood around the reference It ∈ I to be denoised We compute the optical flow between It and every frame Ii ∈ I−Tb,..., It+Tf, which is used to spatially align them For every r×r patch P in It, if it has not been occluded in any of the frames, collecting patches in its trajectory yields a set P of W = Tb+Tf +1 patches with almost-equal intensities but different noise realizations To avoid this averaging producing an incorrectly reconstructed patch due to occluded patches, we replace the occluded patches in P with similar non-occluded ones in their spatio-temporal neighbourhood We then perform a Weighted Principal Component Analysis (WPCA) on Pˆ to reconstruct a filtered version of P by adding detail—but not noise—to b

Performance and Comparison

The optimal proportion α of returned noise before demosaicking should depend on the conditions of the captured scene, where low-light scenes taken with a higher ISO benefit from a lower α—that is, a stronger denoising before demosaicking—as the noise model in darker regions is more reliable in the RAW domain We study the effect of α in more detail in Section IV-B The proposed method shines when it comes to its adaptability to different kinds of sequences and noise levels, which are still a challenge for modern neural networks In comparison with state-of-the-art deep learning based methods, our method outperforms FlexISP, RViDeNet and UDVD in all cases and metrics Our method also surpasses or matches MaskDnGAN in PSNR for ISOs ranging from 3200 to 12800, in SSIM and MS-SSIM for the range 1600 to 6400, and keeps close behind it in VMAF

Conclusion

We have presented a self-similarity-based video denoising scheme that exploits both the data at the camera sensor and information on the RGB domain We have shown that a correct balance between two denoisers—one before and one after demosaicking—significantly improves image quality over denoising only before or after demosaicking The optimal balance depends on the noise level, increasing the RAW filtering for higher ISO values We have also integrated a temporal trajectory prefiltering step before the usual spatio-temporal denoising stage and studied its effects on texture reconstruction We have found that, with strong restrictions on the motion model to avoid misalignments, this step can greatly help in the recovery of fine details, especially at high noise levels The proposed method only requires an estimation of the noise model at the sensor, which can be obtained easily with a simple camera calibration procedure This allows an accurate noise removal at any noise level, making our method suitable for real-world videography The experiments show the competitiveness of our approach against state-of-the-art deep learning based methods in terms of visual quality, and demonstrate that it can better adapt to general video sequences

REFERENCES

2410.02572 1

8

14

3

14

3

5

6

5

--- Page 1 --- A self-similaritybased denoising scheme that weights both a pre- and a postdemosaicking denoiser for Bayer-patterned CFA video data We show that a balance between the two leads to better image quality, and we empirically find that higher noise levels benefit from a higher influence pre-demosaicking We also integrate temporal trajectory prefiltering steps before each denoiser, which further improve texture reconstruction The proposed method only requires an estimation of the noise model at the sensor, accurately adapts to any noise level, and is competitive with the state of the art, making it suitable for real-world videography

--- Page 2 --- Digital photographs and videos, as seen on a screen, are the end result of a complex chain of transformations that process the energy of incident photons in an observed scene into pixel values [1] The goal of these transformations—henceforth referred to as the Image Signal Processing (ISP), or simply the processing pipeline—is to ensure that the original visual information in that scene is faithfully preserved Physical factors such as sensor size, temperature and electronic interference, as well as the random nature of the photoelectric effect and the conversion from electrical signals to digital data make the presence of noise during image acquisition inevitable [4] As noise is usually undesirable, especially when its amplitude obfuscates the details of the scene, denoising is an indispensable step of the ISP Noise is much harder to remove after an image or video has undergone the complete processing pipeline The pixel values of a processed photograph are usually compressed, they have a lower bit depth than the original RAW data, they no longer hold a direct relation with the number of photons, and their noise has become heavily correlated [5] The limitations of denoising after the full processing pipeline are especially pronounced in low-light conditions, where the signal is weaker and noise tends to be more noticeable [6] Darker scenes usually need to be captured with higher camera ISO sensitivity values to be visible, at the expense of amplifying the noise even further Longer exposure times can be used to mitigate this effect, but they can also introduce motion blur, being an unrealistic solution in the case of video These issues have led to the common practice of performing denoising in the RAW domain before demosaicking, where the noise is spatially uncorrelated and the pixel values are still linearly related to the number of photons [5], [7], [8] Despite this, Jin et al. [9] have provided empirical results showing that denoising just after demosaicking may lead to a better texture recovery, on the condition that a stronger filtering is applied Some works have followed that observation and proposed denoising methods in the demosaicked domain [10], [11] The same authors of [9] recently argued in [12] that, for high noise levels, there is a moderate benefit in applying a partial RAW denoising followed by demosaicking and a second denoising on the RGB image The study is conducted using black box optimization with two BM3D [13] denoisers, and applies to synthetic Bayer CFA single image data with added uniform white Gaussian noise The authors conclude that such improvement is too marginal to justify the use of a combined approach and that in practice those noise levels do not appear in real RAW data In the case of RAW video arXiv:2410. p. 0

Where the paper says it

The same authors of [9] recently argued in [12] that, for high noise levels, there is a moderate benefit in applying a partial RAW denoising followed by demosaicking and a second denoising on the RGB image. The study is conducted using black box optimization with two BM3D [13] denoisers, and applies to synthetic Bayer CFA single image data with added uniform white Gaussian noise.

02572v2 [eess. p. 0
Where the paper says it

In the case of RAW video arXiv:2410. 02572v2 [eess.

IV] 29 Nov 2024

--- Page 3 --- DIGITAL photographs and videos, as seen on a screen, are the end result of a complex chain of transformations that process the energy of incident photons in an observed scene into pixel values [1] The goal of these transformations—henceforth referred to as the Image Signal Processing (ISP), or simply the processing pipeline—is to ensure that the original visual information in that scene is faithfully preserved Digital cameras capture images using an electronic image sensor, most commonly Charge Coupled Device (CCD) or Complementary Metal Oxide Semiconductor (CMOS), which is a device that converts light into an electrical signal Light that is captured by the sensor passes through a Color Filter Array (CFA), a patterned layer that allows only a specific wavelength—red, green or blue—to pass through at each pixel Usually, the CFA follows a 2 × 2 Bayer pattern, where the green filters are duplicated to mimic the human eye’s higher sensitivity to green light [2] The values that are read at this stage are called the RAW data, and they are a direct representation of the number of photons that were captured by each pixel [3] The remaining color information must be interpolated—demosaicked—from the surrounding pixels in order to create a full-color image In addition, other transformations are typically applied during the processing pipeline to reconstruct the captured scene: color corrections such as white balance, gamma correction and tone mapping, as well as other adjustments to compensate for lens shading and to sharpen the results Physical factors such as sensor size, temperature and electronic interference, as well as the random nature of the photoelectric effect and the conversion from electrical signals to digital data make the presence of noise during image acquisition inevitable [4] As noise is usually undesirable, especially when its amplitude obfuscates the details of the scene, denoising is an indispensable step of the ISP Noise is much harder to remove after an image or video has undergone the complete processing pipeline The pixel values of a processed photograph are usually compressed, they have a lower bit depth than the original RAW data, they no longer hold a direct relation with the number of photons, and their noise has become heavily correlated [5] The limitations of denoising after the full processing pipeline are especially pronounced in low-light conditions, where the signal is weaker and noise tends to be more noticeable [6] Darker scenes usually need to be captured with higher camera ISO sensitivity values to be visible, at the expense of amplifying the noise even further Longer exposure times can be used to mitigate this effect, but they can also introduce motion blur, being an unrealistic solution in the case of video These issues have led to the common practice of performing denoising in the RAW domain before demosaicking, where the noise is spatially uncorrelated and the pixel values are still linearly related to the number of photons [5], [7], [8] Despite this, Jin et al. [9] have provided empirical results showing that denoising just after demosaicking may lead to a better texture recovery, on the condition that a stronger filtering is applied Some works have followed that observation and proposed denoising methods in the demosaicked domain [10], [11] The same authors of [9] recently argued in [12] that, for high noise levels, there is a moderate benefit in applying a partial RAW denoising followed by demosaicking and a second denoising on the RGB image The study is conducted using black box optimization with two BM3D [13] denoisers, and applies to synthetic Bayer CFA single image data with added uniform white Gaussian noise

--- Page 4 --- representation of the number of photons that were captured by each pixel [3] In addition, other transformations are typically applied during the processing pipeline to reconstruct the captured scene: color corrections such as white balance, gamma correction and tone mapping, as well as other adjustments to compensate for lens shading and to sharpen the results Physical factors such as sensor size, temperature and electronic interference, as well as the random nature of the photoelectric effect and the conversion from electrical signals to digital data make the presence of noise during image acquisition inevitable [4] As noise is usually undesirable, especially when its amplitude obfuscates the details of the scene, denoising is an indispensable step of the ISP Noise is much harder to remove after an image or video has undergone the complete processing pipeline The pixel values of a processed photograph are usually compressed, they have a lower bit depth than the original RAW data, they no longer hold a direct relation with the number of photons, and their noise has become heavily correlated [5] The limitations of denoising after the full processing pipeline are especially pronounced in low-light conditions, where the signal is weaker and noise tends to be more noticeable [6] Darker scenes usually need to be captured with higher camera ISO sensitivity values to be visible, at the expense of amplifying the noise even further Longer exposure times can be used to mitigate this effect, but they can also introduce motion blur, being an unrealistic solution in the case of video These issues have led to the common practice of performing denoising in the RAW domain before demosaicking, where the noise is spatially uncorrelated and the pixel values are still linearly related to the number of photons [5], [7], [8] Despite this, Jin et al. [9] have provided empirical results showing that denoising just after demosaicking may lead to a better texture recovery, on the condition that a stronger filtering is applied Some works have followed that observation and proposed denoising methods in the demosaicked domain [10], [11] The same authors of [9] recently argued in [12] that, for high noise levels, there is a moderate benefit in applying a partial RAW denoising followed by demosaicking and a second denoising on the RGB image The study is conducted using black box optimization with two BM3D [13] denoisers, and applies to synthetic Bayer CFA single image data with added uniform white Gaussian noise

--- Page 5 --- representation of the number of photons that were captured by each pixel [3] In addition, other transformations are typically applied during the processing pipeline to reconstruct the captured scene: color corrections such as white balance, gamma correction and tone mapping, as well as other adjustments to compensate for lens shading and to sharpen the results Physical factors such as sensor size, temperature and electronic interference, as well as the random nature of the photoelectric effect and the conversion from electrical signals to digital data make the presence of noise during image acquisition inevitable [4] As noise is usually undesirable, especially when its amplitude obfuscates the details of the scene, denoising is an indispensable step of the ISP Noise is much harder to remove after an image or video has undergone the complete processing pipeline The pixel values of a processed photograph are usually compressed, they have a lower bit depth than the original RAW data, they no longer hold a direct relation with the number of photons, and their noise has become heavily correlated [5] The limitations of denoising after the full processing pipeline are especially pronounced in low-light conditions, where the signal is weaker and noise ten…

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Optimal balance between denoisers can be determined by scene conditions; "Low-light scenes taken with a higher ISO benefit from a lower α—that is, a stronger denoising before demosaicking—as the noise model in darker regions is more reliable in the RAW domain and the correlated noise after demosaicking can be hard to remove due to the lower signal-to-noise ratio."

  2. Integration of temporal trajectory prefiltering improves texture reconstruction by estimating the trajectories of small overlapping patches and filters their contents, permitting a significant reduction of the noise without producing any texture oversmoothing, which is particularly beneficial when if an object on a sequence is difficult to track (i.e. it moves fast), an erroneous optical flow estimation may cause reconstruction inaccuracies.

  3. The system can adapt to varying noise levels by controlling the RAW filtering degree based on ISO, as the method only requires an estimation of the noise model at the sensor, accurately adapts to any noise level.

Abstract

Denoising is one of the fundamental steps of the processing pipeline that converts data captured by a camera sensor into a display-ready image or video. It is generally performed early in the pipeline, usually before demosaicking, although studies swapping their order or even conducting them jointly have been proposed. With the advent of deep learning, the quality of denoising algorithms has steadily increased. Even so, modern neural networks still have a hard time adapting to new noise levels and scenes, which is indispensable for real-world applications. With those in mind, we propose a self-similarity-based denoising scheme that weights both a pre- and a post-demosaicking denoiser for Bayer-patterned CFA video data. We show that a balance between the two leads to better image quality, and we empirically find that higher noise levels benefit from a higher influence pre-demosaicking. We also integrate temporal trajectory prefiltering steps before each denoiser, which further improve texture reconstruction. The proposed method only requires an estimation of the noise model at the sensor, accurately adapts to any noise level, and is competitive with the state of the art, making it suitable for real-world videography.

Related papers