R cubed: Reconstruction, Raw, and Rain: Deraining Directly in the Bayer Domain

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "R cubed: Reconstruction, Raw, and Rain: Deraining Directly in the Bayer Domain".

Jane: Here's the gist of the paper. The gist: Learning directly on raw Bayer mosaics yields superior reconstructions by avoiding irreversible losses introduced by post-ISP processing, which advocates for an ISP-last paradigm for low-level vision and opens the door to end-to-end learnable camera pipelines<ref:2509.24022#pg2> Significance of Raw Domain Reconstruction Image reconstruction from corrupted images is crucial across many domains, and this paper uses the rain degradation problem as a ”use case” to show that these losses are avoidable and show that learning directly on raw Bayer mosaics yields superior reconstructions<ref:2509.24022#pg2> The ISP pipeline is designed to produce visually pleasing images for human consumption, but from a restoration standpoint, it is a lossy and irreversible transformation<ref:2509.24022#pg2> This pipeline involves demosaicing which introduces interpolation artifacts and blends spatially localized features<ref:2509.24022#pg2> Furthermore, white balancing and color correction alter pixel intensities based on learned or estimated illumination models, often distorting the true spectral distribution<ref:2509.24022#pg2> Consequently, reconstruction in the sRGB (we use sRGB and RGB interchangeably, all work in the paper is done in the sRGB space) domain becomes an ill-posed problem<ref:2509.24022#pg2> Bayer images represent the closest approximation to the original sensor measurement, preserving spatial and photometric information<ref:2509.24022#pg2> Each pixel corresponds to a direct photo-detection measurement with minimal transformation, offering a linear response to scene illumination and full access to high-frequency information before it is smoothed or distorted by the ISP<ref:2509.24022#pg2> Evaluation of Reconstruction Pipelines The study evaluates both post-ISP and Bayer reconstruction pipelines<ref:2509.24022#pg2> The software ISP pipeline simulates key operations typically performed in camera hardware, including black level subtraction, demosaicing, lens shading, white balancing, color correction, global tone mapping, local tone mapping, and gamma<ref:2509.24022#pg2> This pipeline ensures that Bayer/sRGB images fed into the deraining network preserve photometric properties comparable to standard camera outputs<ref:2509.24022#pg2> The model architecture is designed to investigate the impact of performing deraining in the raw sensor space against the sRGB space<ref:2509.24022#pg2> Three architectures are proposed, including UNET, MONOCULAR UNET, and CBAM, which follow a similar rough design<ref:2509.24022#pg2> The input to the models are original images, sRGB (B,3,H,W) or Bayer (B,1,H,W), and the model outputs the input image reconstructed (derained) of same dimensions<ref:2509.24022#pg2> Novel Metric for Fidelity Assessment Conventional full-reference metrics—PSNR, SSIM, and LPIPS—assume that the reference and the reconstruction live in exactly the same radiometric space<ref:2509.24022#pg2> This assumption collapses in deraining because heavy rain corrupts the statistics that drive the camera image-signal-processing (ISP) stack<ref:2509.24022#pg2> The paper introduces the Information Conservation Score (ICS), a color-invariant metric that aligns more closely with human opinion than PSNR or SSIM<ref:2509.24022#pg2> ICS couples a color-agnostic multi-scale SSIM term with a frequency-matching KL-divergence<ref:2509.24022#pg2> The proposed Information Conservation Score (ICS) Eq.

Tom: 1 combines perceptual similarity with frequencydomain fidelity: ICS(X, &hat X) = &lambda · MS-SSIM(X, &hat X) + (1 - &lambda) · D KL(P X(u,v,w) P&hat X(u,v,w))<ref:2509.24022#pg2> This dual-domain formulation enables ICS to penalize not only spatial inconsistencies but also frequency-domain distortions and spectral energy loss<ref:2509.24022#pg2> The spectral KL component of ICS acts as an information–theoretic safeguard against over-smoothing, quantifying the expected penalty for discarding frequency content that is critical for sharp, rain-free reconstruction<ref:2509.24022#pg2> Dataset and Experimental Results To support the study on deraining in raw and sRGB spaces, a custom stereo dataset was collected using two synchronized FLIR Blackfly S 2.8MP cameras<ref:2509.24022#pg2> The rig consists of two synchronized FLIR Blackfly S 2.8MP cameras, each capable of capturing raw Bayer images<ref:2509.24022#pg2> Data collection was carried out in a range of realistic scenarios, including sessions with and without windshield wipers operating at different speeds<ref:2509.24022#pg2> The dataset includes a total of 89 diverse training scenes, 30 identity training scenes, 10 held-out test scenes, and 15 validation scenes (no GT)<ref:2509.24022#pg2> On the test split, the raw-domain model improves sRGB results by up to +0.99 dB PSNR and +1.2 % ICS, while running faster with lower GFLOPs<ref:2509.24022#pg2> Table 3 shows that for Scene 6, the CBAM Bayer model achieves a higher ICS of 0.864 compared to the RGB Model's 0.852<ref:2509.24022#pg2> These findings advocate an ISP-last paradigm for lowlevel vision: preserve raw sensor data until the final stage, then apply a learnable or task-aware ISP<ref:2509.24022#pg2> The study concludes that sensor-level deraining—learning directly on raw Bayer mosaics outperforms conventional postISP approaches<ref:2509.24022#pg2> Contributions of the Work The contributions of this paper are threefold<ref:2509.24022#pg2> 1.

Jane: We demonstrate the significance of reconstruction in the Bayer domain over the post ISP domain<ref:2509.24022#pg2> 3.

Tom: We introduce a novel dataset comprising temporally synchronized dual-camera video sequences captured in both Bayer (raw) and sRGB (post-ISP) formats<ref:2509.24022#pg2> 4.

Title and authors: Tom: So we're looking at this paper called "R cubed: Reconstruction, Raw, and Rain: Deraining Directly in the Bayer Domain". Jane, what’s the main angle here?

Jane: Well, it really zeroes in on how much information you lose when you process an image through a standard camera pipeline versus if you work directly with the raw data. The core idea is that learning directly on those raw Bayer mosaics gives better results for reconstructing images corrupted by rain than training on images already processed through an ISP.

Lu: It’s about moving away from the post-ISP paradigm for low-level vision tasks, which is where most of our current models are trained. The authors show that the ISP pipeline introduces irreversible losses, like demosaicing artifacts and color correction distortions, which makes reconstruction in the sRGB domain an ill-posed problem.

Meng: Irreversible losses are a big deal for us because when we train on post-ISP data, we’re essentially training our models to correct for those errors instead of learning the true underlying signal from the sensor itself. That sounds like it could lead to more robust models, but it also makes deployment harder if you need to go back and fix that pipeline later.

Lalam: I've reviewed the literature, and this paper directly addresses that fundamental limitation by proposing a way to avoid those losses entirely by learning on the raw Bayer data itself.

Tom: Right. So we’re talking about avoiding demosaicing artifacts and illumination model distortions by staying in the raw sensor space for deraining. What does that mean practically for us when we think about building these camera pipelines?

Jane: It means you can design end-to-end learnable camera pipelines, where the entire process, from raw sensor input to final output, is learned together. Instead of a fixed series of steps that lose fidelity at each stage, you have one big model that learns the best way to recover the original scene from the raw noise.

Lu: The authors propose three different model architectures to see what works best when operating in either raw sensor space or sRGB space, including UNET, MONOCULAR UNET, and CBAM. They are testing whether performing deraining directly on the raw sensor data yields superior reconstructions compared to doing it in the sRGB space.

Meng: I’m interested in those architectures because if one of them is better at handling the noise and structure in that raw domain, it could mean we can simplify our downstream processing significantly. But what about the practical side—how much computational overhead are we looking at?

Tom: The paper mentions that even a very small U-Net benefits from operating on raw Bayer data, which suggests this might be more feasible for resource-constrained hardware than some of the larger models we typically deploy.

Title and authors: Jane: And they don't just stop at reconstruction. They introduce a new way to measure how good the reconstruction actually is, which is super important because traditional metrics fall apart when you move between raw and processed spaces.

Lu: They introduce something called the Information Conservation Score, or ICS, which they say couples a color-agnostic multi-scale SSIM term with a frequency-matching KL-divergence. It’s designed to penalize not just spatial inconsistencies but also spectral energy loss and frequency distortions.

Meng: A dual formulation that includes the spectral KL component sounds like it’s trying to safeguard against over-smoothing, which is something I worry about when we try to aggressively denoise images for speed. How much information conservation does this ICS actually show compared to PSNR or SSIM?

Tom: The results on the test set are quite strong; the raw-domain model improves sRGB results by up to zero point nine nine dB PSNR and one point two percent ICS, while also running faster with lower GFLOPs than some of the post-ISP methods they compared it against.

Jane: That improvement in ICS is significant because it aligns more closely with what people actually see as good quality, rather than just matching a pixel value perfectly, which is what PSNR does. It suggests that even if the PSNR number looks lower, the visual quality is better because the frequency content—the sharp details—is preserved.

Lu: The dataset they built to test this was custom stereo data from two synchronized FLIR Blackfly S 2 point 8MP cameras, capturing both raw Bayer and sRGB formats in realistic rain scenarios, including sessions with and without windshield wipers operating at different speeds <ref:2509.24022#pg3>. They created the RAW-RAIN benchmark for this.

Meng: A custom dataset sounds promising for reproducibility, but I need to know how scalable this is. Are we talking about training on eighty-nine diverse scenes plus validation sets? And what about the challenge of getting that kind of synchronized data reliably in a production environment?

Tom: They curated a total of eighty-nine diverse training scenes, thirty identity training scenes, and held-out test and validation splits. The paper published the first public set of paired twelve-bit Bayer and bit-depth matched sRGB rain scenes to enable reproducible research on pre-ISP restoration <ref:2509.24022#pg1,12-bit Bayer and bit-depth matched sRGB>.

Jane: That public release is a big thing because it lets everyone check if their own models are performing well on this specific type of data, which is crucial for moving from research to real world application. So, what’s the bigger picture here?

Lu: The authors conclude by advocating strongly for an ISP-last paradigm for low-level vision: preserve raw sensor data until the final stage, then apply a learnable or task-aware ISP. They show that sensor-level deraining directly on raw Bayer mosaics outperforms conventional postISP approaches.

Title and authors: Meng: So the implication is that we should be pushing research toward pre-ISP processing, treating the sensor output as our primary source of truth before any artistic or aesthetic adjustments are made by a learned pipeline. That shifts where we focus our engineering efforts for low-level vision tasks.

Tom: Exactly. It’s about making sure we capture the highest fidelity signal possible before we introduce those irreversible transformations that happen in the ISP pipeline, which is what makes these problems ill-posed in the sRGB domain.

Jane: It moves us toward building camera pipelines that are fundamentally more faithful to what the sensor measured, and that raw data preservation leads to a more accurate understanding of the scene being reconstructed. So, we’ve seen how this paper uses rain degradation as a use case to show these losses are avoidable.

Lu: And they’re opening the door for end-to-end learnable camera pipelines where the whole process is optimized together, not just isolated components. It suggests that even a very small U-Net can benefit from operating on raw Bayer data.

Meng: For deployment, that single-channel input requirement for the Bayer domain pipeline is a nice bonus because it keeps the complexity down when we have to run this on hardware that isn't top tier. But what about scaling up that model if we need to handle more complex scenes?

Tom: The paper shows consistency across their test set, where the ICS consistently shows greater information conservation when deraining in the Bayer domain compared to sRGB processing. That’s a strong signal for deployment choices based on fidelity versus speed trade-offs.

Jane: So, to wrap up on "R cubed: Reconstruction, Raw, and Rain: Deraining Directly in the Bayer Domain," this paper really advocates that for high-fidelity restoration tasks like deraining, we need to stop treating the ISP output as our starting point and start treating the raw sensor data as the most valuable input.

Lu: It provides a new benchmark for testing these concepts, RAW-RAIN, and a new metric, ICS, that gives us a better way to judge if we’re actually saving information or just tweaking pixels around.

Meng: From an engineering standpoint, it tells us that the architectural choices matter more than just picking the biggest model; we need models designed for that raw domain input if we want to maximize signal preservation on a budget.

Tom: So the message is clear: focus on pre-ISP processing for low-level vision, use metrics like ICS to guide your design, and learn directly from the raw Bayer mosaics if you want superior reconstructions in challenging conditions like rain. That’s where we’ll leave it for today.

The paper's summary: Tom: So, we’re looking at the summary of "R cubed: Reconstruction, Raw, and Rain: Deraining Directly in the Bayer Domain". Basically, they're saying that when you have rain corruption in an image—that's your input—the best way to fix it isn't to clean up the final sRGB picture.

Jane: Right. They argue that the real problem is that every time you go through a standard camera pipeline, like demosaicing or color correction, you introduce losses that are basically irreversible. So, if you start your AI model right on those raw Bayer mosaics from the sensor—the actual measurement—you avoid all those mistakes because you haven't messed with the data yet.

Lu: That means they’re showing that learning directly on the raw Bayer data is superior for reconstruction compared to training on images that have already been processed through a standard ISP pipeline. They use rain degradation as their test case, and their results show an improvement in both PSNR and this new metric called the Information Conservation Score.

Meng: The Information Conservation Score sounds interesting because it tries to measure if you’re just matching pixels perfectly, or if you’re actually saving the important information that makes an image look sharp. If the ICS is higher, they think you've kept more of what matters for a good reconstruction.

Tom: Exactly. And they even show that this raw-domain approach can be faster on some hardware because the models don't have to deal with all those extra noise and artifacts introduced by the ISP pipeline upfront. It suggests a whole shift in how we design low-level vision systems, moving toward an ISP-last paradigm where you fix it later.

Jane: What this means for us as people who just watch a show or drive? It means the technology that makes our images look clear in bad weather—like rain—needs to be built differently. Instead of relying on post-processing fixes, we need systems that are inherently designed to handle the raw data better from the start.

Lu: They also created a new dataset called RAW-RAIN which is important because it lets other researchers actually test this idea on real, synchronized video captured in both raw and processed formats. It's a public set for reproducible research.

Meng: A custom dataset sounds good for testing, but the challenge is getting that kind of perfectly synchronized data consistently in the real world to train something robust enough to handle everything. It’s not just about having a big number of scenes; it’s about getting realistic movement and lighting captured correctly across both formats.

Tom: So they're telling us that for serious restoration work, we need to focus on pre-ISP processing and use metrics like the ICS to make sure our models are actually preserving signal, not just fooling us with a high PSNR number.

Jane: And this opens the door for building end-to-end learnable camera pipelines where everything is optimized together from the raw sensor input all the way to the final output. It’s about treating the whole vision system as one learning unit.

Lu: That’s what they are aiming for, and their findings suggest that even a small model architecture like a U-Net can perform better when it's trained on that raw Bayer data because it has cleaner input to start with.

Meng: If we adopt this, we need to think about how easy it is to swap out the ISP part of the pipeline later, rather than having everything baked into one massive pre-ISP model. That’s a real deployment consideration.

Tom: Exactly. The conclusion is that for high-fidelity restoration tasks like deraining, sticking to the raw sensor data input and using information-theoretic metrics like ICS gives you better results and better signal preservation than working in the processed sRGB space.

Jane: So we’ve seen how this paper uses rain degradation as a use case to show that avoiding those post-ISP losses is key, and it's setting up the path for a whole new way to design camera pipelines for low-level vision.

The paper's improvements: Tom: So we're talking about what these authors suggest as improvements for their raw domain deraining work in "R cubed". They aren't just presenting one model; they are showing how this approach changes the entire way we think about camera restoration.

Jane: Right. Instead of just saying "this works," they are pushing the idea that we should adopt an ISP-last paradigm for low-level vision tasks altogether. It means preserving the raw sensor data until the very last possible moment before applying a learned or task-aware ISP later on.

Lu: They’re suggesting that this isn't just a fix for rain; it’s a fundamental change in how we handle image restoration problems, showing that working directly on the Bayer domain is more robust than trying to correct distortions in the sRGB space first.

Meng: If they are pushing an ISP-last paradigm, what does that mean for us shipping models right now? It means we can design our models to be more flexible and less tied down by a fixed, lossy pipeline. It allows for end-to-end learnable camera pipelines where the entire process is optimized as one system.

Tom: It means that even very small architectures, like a standard U-Net, can benefit from operating on raw Bayer data because the input signal is much cleaner than what you get after demosaicing and color correction. That’s a big mechanical win for deployment constraints.

Jane: And they are suggesting that we need new metrics, like the Information Conservation Score, not just PSNR, because those metrics give us a better picture of actual information preservation versus just matching pixel values that might look good but aren't actually accurate.

Lu: They’re showing that the ICS consistently shows greater information conservation across their test set when deraining in the raw Bayer domain compared to processing it in sRGB space, which is strong evidence for this direction.

Meng: So, if we want to build systems that handle real-world noise and rain without breaking at scale, the direction is clear: focus on pre-ISP processing and using metrics like ICS to guide our design choices. That’s what I can deploy reliably.

Tom: Exactly. The implication is that we should shift our engineering efforts toward designing models that are inherently more faithful to the sensor's original measurement, treating the raw Bayer data as the most valuable input from the start.

Jane: And this moves us toward building camera pipelines that are fundamentally more faithful to what the sensor measured before any aesthetic adjustments happen. It’s about capturing higher fidelity signal right at its source.

Lu: They also published a public set of paired twelve-bit Bayer and bit-depth matched sRGB rain scenes, which is crucial because it lets other researchers actually reproduce this kind of pre-ISP restoration work reliably.

Meng: That public release is important for validation, but the real test will be how well these models perform when we run them on actual traffic data, not just these curated training scenes.

Tom: So to sum up the improvements: it’s about adopting an ISP-last paradigm, using metrics like ICS to guide model design, and learning directly from raw Bayer mosaics for superior reconstruction quality in bad conditions.

Jane: This sets a new standard for low-level vision tasks, showing that preserving raw sensor data is the way to go when dealing with complex corruption like rain.

Conclusion: Tom: So we’re wrapping up on "R cubed: Reconstruction, Raw, and Rain: Deraining Directly in the Bayer Domain." Basically, they've shown that learning directly on raw Bayer mosaics gives superior reconstructions for deraining compared to post-ISP methods.

Jane: It really boils down to this idea that you should be preserving the raw sensor data until the very last stage before applying any learned or task-aware ISP pipeline. It’s a big shift in how we think about low-level vision tasks.

Lu: They’re advocating for an ISP-last paradigm for low-level vision, arguing that this approach is fundamentally more robust against the irreversible losses introduced by standard image processing steps like demosaicing.

Meng: I think the main implication for deployment is that we need to prioritize pre-ISP processing, and using metrics like the Information Conservation Score to guide our design choices if we want to maximize signal preservation on a budget.

Lalam: I’m confident this direction is correct because their results show that sensor-level deraining directly on raw Bayer mosaics outperforms conventional post-ISP approaches across their test set.

Tom: It means we should be pushing research toward pre-ISP processing for low-level vision, treating the sensor output as our primary source of truth before any aesthetic adjustments are made by a learned pipeline.

Jane: Exactly. This opens the door for building end-to-end learnable camera pipelines where the whole process is optimized together, not just isolated components.

Lu: And they’ve given us a new benchmark, RAW-RAIN, which is important because it lets other researchers actually reproduce this kind of pre-ISP restoration work across different sensor fidelities.

Meng: A custom dataset sounds good for testing, but the challenge is getting that kind of perfectly synchronized data consistently in the real world to train something robust enough to handle everything.

Tom: So the message is clear: focus on pre-ISP processing for low-level vision, use metrics like ICS to guide your design, and learn directly from the raw Bayer mosaics if you want superior reconstructions in challenging conditions like rain.

Jane: This paper uses rain degradation as a use case to show that avoiding those post-ISP losses is key, and it's setting up the path for a whole new way to design camera pipelines for low-level vision.