Lighting-aware Unified Model for Instance Segmentation

arXiv:2605.20436 · cs.CV · Submitted 2026-05-19 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "Lighting-aware Unified Model for Instance Segmentation".

Jane: Here's the gist of the paper. Foundation models like the Segment Anything Model (SAM) demonstrate impressive zero-shot generalization but frequently degrade under diverse real-world illumination, particularly for instance segmentation.

Tom: This work addresses this limitation by developing Lighting Convolutional-Attention (LCA), an adapter module that enhances segmentation robustness without fine-tuning the heavy backbone.

Jane: Extensive experimental results demonstrate that our approach successfully bridges the domain gap, delivering superior lighting-robust segmentation.

Tom: Pruning redundant reasoning tokens The authors propose Pairwise Lighting Augmentation Pipeline with Lighting Convolutional Attention (PLAP-LCA), a framework that equips the Segment Anything Model (SAM) with robust lighting invariance without requiring computationally prohibitive fine-tuning of its heavy backbone.

Title and authors: Tom: So we're looking at this paper now, "Lighting-aware Unified Model for Instance Segmentation," and it addresses a problem where foundation models like SAM just start failing when the lighting changes—like going from bright daylight to a dark room.

Jane: Exactly. The authors are proposing this new thing called Lighting Convolutional-Attention, or LCA, which is an adapter they put onto the frozen encoder of SAM to make it more robust to different lighting conditions without having to retrain the whole heavy model.

Lu: What's interesting here is their idea that you need three levels of intervention: data level, architecture level, and training level—they argue that just fixing the data or just tweaking the model isn't enough for lighting invariance.

Meng: From a deployment standpoint, I wonder how much overhead this LCA module adds to the running cost when we have to deploy this on our production systems.

Lalam: My analysis shows that the core idea is using a dual-branch architecture within LCA that separates standard spatial features from those illumination sensitive structural edge pieces, which they get from Laplacian filtered contrast maps.

Tom: That sounds mechanically interesting because they are explicitly trying to isolate what’s structural versus what’s just a lighting artifact, and they use that contrast information to guide the attention.

Jane: They build their training signal using a curriculum based pairwise strategy, meaning they train against both clean images and their variants using a combination of standard segmentation loss and this new feature consistency loss.

Lu: The consistency loss specifically penalizes discrepancies between the sigmoid activated outputs of these two streams with an L1 penalty, which is designed to force the network to learn representations that are invariant to those lighting changes.

Meng: So they're training it on pairs of images, clean and variant, using a loss function that makes them agree on the mask output despite the lighting shift; that’s how they ensure robustness.

Tom: And for illumination awareness, they use three parallel attention gates—channel attention, spatial attention, and contrast attention—with the contrast gate specifically detecting lighting edges by projecting features to a single channel and applying a fixed Laplacian kernel.

Jane: That contrast gate is key because it directly targets those structural changes we care about, not just the overall brightness of the image.

Lu: Their empirical results on Cityscapes show that LCA improves mean IoU from zero point two three five to zero point three eight zero, which is a big jump of sixty one point seven percent over the SAM baseline under lighting variant conditions.

Meng: Sixty one percent improvement is substantial, but we need to see how this scales when we move from Cityscapes to something like a real traffic scene with much more complex shadows and glare.

Tom: They also show that in really challenging cases, LCA can recover a mask around a small low contrast target where the baseline was seeing almost nothing at all—a zero point zero seven IoU compared to their five eighty five for LCA.

Jane: That recovery on that small object is really telling; it suggests the method captures local lighting gradients and penumbra structures effectively through that depthwise separable residual projection.

Lu: The total trainable parameter count for the LCA plus the decoder variant is six point four million, which is only two point six percent overhead compared to a SAM ViT-B baseline, which makes it quite light for deployment.

Meng: Six point four million parameters is manageable, but I still have to worry about that training requires paired clean and variant images, which sounds like a lot of data preparation work upfront.

Tom: Right, the authors admit that training needs those paired clean variant images, and they also mentioned a couple of limitations. One is that they haven't tested extreme lighting conditions like complete darkness with point lights yet.

Jane: And another thing they flagged is the computational cost of the LCA module itself, which could hinder edge deployment if we need super low latency.

Lu: They also noted that their conflict sampling relies on manually defined operation groups, and they only focused on instance segmentation without looking at how this extends to video or panoptic tasks.

Meng: So they're making a strong case for instance segmentation robustness but leaving the door open for other modalities that need similar lighting awareness.

Tom: Overall, the paper concludes that this PLAP-LCA framework helps achieve superior lighting-robust segmentation by combining lighting aware feature adaptation with decoder tuning, and it maintains stable performance across a wide range of consistency loss weights.

Jane: So to wrap up, this work shows how you can make foundation models more reliable in real world scenarios—like nighttime driving or medical imaging—by adding these specific attention mechanisms that reason about structure instead of just reacting to the light.

Lu: The main implication is that we don't have to do expensive full backbone fine-tuning; we can inject this lightweight adapter into frozen encoders and get significant improvements in segmentation accuracy under varied illumination.

Meng: For us in production, it means a potentially lighter model that still gives us much better performance when things get dark or washed out, assuming we can find those training pairs.

Lalam: I'm quite confident in the results because they show stability across the loss weights, which means the method is pretty reliable as long as you have decent paired data to start with.

Tom: It’s a solid contribution to making these powerful segmentation models more practical for real world use, and we'll keep an eye on how others build on this lighting aware unified model.

The paper's summary: Tom: So, to recap, this paper is about taking models like SAM and making them work in bad light—like at night or in a dark room—by adding an adapter module called LCA that focuses on structural features instead of just the brightness of the image.

Jane: Exactly. They’re essentially building a filter into the model that helps it see what's actually there, like edges and shapes, even when the lighting is terrible. It’s not retraining the whole massive model; it's just plugging in this smaller piece called LCA that sits on top of it.

Tom: The big takeaway is that they managed to bridge a gap between models trained only on perfect data and how we actually see things in the real world, especially for instance segmentation.

Jane: It shows that you don't always need a completely new model or a massive amount of new data to make AI more reliable in tricky situations like nighttime driving or even medical imaging where lighting shifts constantly.

Tom: The results they got are pretty telling on Cityscapes, showing a jump in accuracy, from around zero point two three five mean IoU up to zero point three eight zero under those tough lighting conditions. That’s a sixty one point seven percent improvement compared to the baseline model.

Jane: And it's not just about the big number; they showed that this LCA module can actually pull a mask back into view around small, low contrast objects where the original model was failing completely—going from almost zero IoU up to zero point five eight five in those difficult cases.

Tom: That ability to recover structure in low-signal situations is what makes me excited. It suggests that the way they designed this dual-branch architecture and those three attention gates—channel, spatial, and contrast—is genuinely picking up on physical properties of light changes, not just random noise.

Jane: Because they use a contrast attention gate specifically to look for those lighting edges by projecting the features down into a single channel with a Laplacian kernel; that’s how they explicitly tell the model where the structure is changing due to light.

Tom: And from an engineering standpoint, it's efficient too; the total trainable parameter count for this whole LCA plus decoder setup is six point four million, which is only about two point six percent overhead compared to the original SAM ViT-B baseline. That’s a very lightweight adapter.

Jane: But there are some caveats they mentioned. They said they haven't really tested extreme lighting yet, like total darkness with just a single bright light source, and that since they need clean and variant pairs for training, it means you have to spend time creating those paired datasets first.

Tom: Right, the paper is clear on those limitations; it’s not a perfect solution for every single lighting scenario right now. But the core idea of injecting this feature adaptation module into a frozen encoder to gain robustness without retraining everything is pretty compelling, especially for applications where data collection and fine-tuning are too expensive.

Jane: So, what does this mean for us? It means we can potentially make autonomous driving safer at night or get more reliable results in medical imaging under varying clinical lights without having to rebuild the entire foundation model from scratch.

Tom: Exactly. It makes lighting robustness accessible for resource constrained applications because it's an adapter, not a whole new system overhaul. Now, if you want to see how they specifically built that training signal using the pairwise strategy and the consistency loss function, we’ll take a look at their methodology next.

The paper's improvements: Tom: So, we're looking at how they actually improve things beyond just getting a slightly better score on Cityscapes—they’re talking about three main areas where this approach makes a real difference for deployment.

Jane: They focus first on lighting-invariant feature extraction, which means the model stops getting confused by shadows and glare because it processes RGB features along with contrast maps and Laplacian filtered maps to focus on structure.

Tom: That's what I like; it’s physically motivated sensitivity to structural changes instead of just reacting to how bright the image is, which is a huge mechanical improvement for robustness.

Jane: Then there's the robust training strategy, where they introduce that targeted loss term that explicitly penalizes any discrepancies between clean images and their lighting variants, forcing the model to learn representations that are invariant to those changes.

Tom: That loss function is clever because it actively teaches the network what lighting invariance looks like mathematically, not just hoping it learns it by seeing enough data.

Jane: And they also have this adaptive attention mechanism using three parallel gates—channel, spatial, and contrast—to help the model selectively focus on the most relevant structural information in the input features.

Tom: That contrast attention gate is where they explicitly detect those lighting edges we care about by projecting features to a single channel and applying a fixed Laplacian kernel; that seems like it’s really isolating the light artifact from the shape.

Jane: It changes how the model reasons about what to look at spatially, determining where to attend, while also figuring out which feature channels matter most under different lighting conditions.

Tom: The implication for real-world use is that this architecture lets us deploy models on edge devices that need to handle things like nighttime scenes or indoor low-light environments reliably, without needing a full retraining cycle.

Jane: It means we can achieve superior lighting-robust segmentation in scenarios where the baseline model would completely break down because it wasn't trained for those specific conditions.

Tom: They also highlight the synthetic data generation side, where they use a novel Unity engine to create physically rendered images with pixel perfect annotations, which helps them train this way without needing to collect millions of real-world, paired lighting datasets.

Jane: So what we're seeing here is a unified model that combines this lighting-aware feature processing and decoder tuning for better accuracy *and* better invariance simultaneously.

Tom: They found that combining the LCA module with the decoder variant actually delivers the best overall performance across several different datasets, and their parameter count stays relatively low at six point four million compared to a SAM ViT-B baseline.

Jane: That stability is interesting too; they showed that even with different weights for that consistency loss, like lambda s being varied, the performance only changes by less than zero point one five percent in mean IoU.

Tom: It’s a very stable result, which tells me the mechanism isn't just a lucky fluke on one specific data distribution; it seems fundamentally sound across different lighting variations.

Jane: So what this means for you listening right now is that if you are building an application—whether it’s autonomous systems or medical imaging—you have a way to make your AI more dependable when the conditions change unexpectedly.

Tom: It moves the goalpost from needing perfect training data to building models that can intelligently adapt their feature processing based on the lighting they encounter in real-time.

Jane: We’ll get into how they built those training pairs and the consistency loss function next, because understanding how they force that invariance is key to seeing exactly how much of this improvement is coming from their training signal.

Conclusion: Tom: So, to wrap up, this paper is about developing Lighting-aware Unified Model for Instance Segmentation by adding an LCA adapter that makes models robust to bad lighting conditions without needing a full fine-tune of the heavy backbone.

Jane: Exactly. It shows we can make foundation models much more reliable in real world scenarios like nighttime driving or indoor low light, simply by injecting this lightweight adapter.

Tom: The main implication is that we don't have to spend all that time and money collecting massive paired datasets just to fix lighting issues; we can use this mechanism for a quick, targeted improvement.

Jane: And the results on Cityscapes are pretty strong, showing that the mean IoU went up from zero point two three five to zero point three eight zero under those difficult lighting conditions.

Tom: That sixty one point seven percent improvement is solid because it came from a mechanism that’s actually picking up on structural information through contrast maps and attention gates.

Jane: It shows that this approach works well for improving segmentation accuracy while simultaneously enhancing the model's invariance to illumination shifts across globally, locally, and structurally defined dimensions.

Tom: The parameter count is manageable at six point four million total trainable parameters for the LCA plus decoder variant, which is good news if we’re thinking about deployment on resource constrained systems.

Jane: They also made sure to flag that this method requires paired clean and variant images for training, and they haven't fully tested extreme lighting conditions like total darkness yet.

Tom: Right, the authors are transparent about those limitations; it’s not a universal fix for every single possible light scenario just yet.

Jane: But the core idea is that we can make these powerful models more dependable in scenarios where they're currently failing because of poor lighting conditions, and that’s a practical step forward.

Tom: This Lighting-aware Unified Model for Instance Segmentation provides a clear path for making segmentation more accessible in tough visual environments.

Jane: I think the next big question is how we can extend this idea to other tasks, like video understanding or even panoptic segmentation, because instance segmentation is just one part of the picture.

Tom: Yeah, that’s where we go next; we need to see if this structural attention concept applies equally well when you’re dealing with temporal sequences or scene-level understanding.

Jane: We'll definitely be looking at that; I think seeing how they apply this contrast mechanism to video might open up a lot of new possibilities for real time applications.

Tom: Let's see what the next paper brings to the table, because this work really proves that lighting awareness is a viable and efficient way to boost model performance.

Jane: Agreed; we’ll keep an eye on how they build on this concept, because making AI robust across different visual qualities is going to be a huge focus in the coming years.

Tom: That's all for this session on the Lighting-aware Unified Model for Instance Segmentation; we'll be right back after the break with more research from arXiv.

Iowa State University

cs.CV

Submitted: 2026-05-19

Updated: 2026-11-09

Comments: Accepted to WACV 2027

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 92/100

The gist: Foundation models like the Segment Anything Model (SAM) demonstrate impressive zero-shot generalization but frequently degrade under diverse real-world illumination…

Key concepts

Lighting Convolutional-Attention (LCA)
LCA is an adapter module added to the frozen SAM encoder. It uses a dual-branch design that separates standard spatial features from illumination-sensitive structural edge information derived from Laplacian-filtered contrast maps. This allows the model to focus on physical structures instead of lighting effects.
Pairwise Training Strategy
This training method optimizes the model using two losses: standard segmentation loss (BCE and Dice) and a consistency loss. The consistency loss penalizes differences between the network's output on a clean image and its output on an illumination variant, forcing the model to learn lighting-invariant representations.
Contrast Attention Gate
This specific attention gate detects lighting edges by projecting features into a single-channel intensity map and applying a fixed Laplacian kernel. It is crucial for explicitly capturing structural information related to lighting changes, helping the network distinguish between true objects and illumination artifacts.

Terminology

Summary

Foundation models like the Segment Anything Model (SAM) demonstrate impressive zero-shot generalization but frequently degrade under diverse real-world illumination, particularly for instance segmentation. p. 0

Where the paper says it

edu Abstract. Foundation models like the Segment Anything Model (SAM) demonstrate impressive zero-shot generalization but frequently degrade under diverse real-world illumination, particularly for instance segmentation.

p. 0
Where the paper says it

edu Abstract. Foundation models like the Segment Anything Model (SAM) demonstrate impressive zero-shot generalization but frequently degrade under diverse real-world illumination, particularly for instance segmentation.

This work addresses this limitation by developing Lighting Convolutional-Attention (LCA), an adapter module that enhances segmentation robustness without fine-tuning the heavy backbone. p. 0
Where the paper says it

Foundation models like the Segment Anything Model (SAM) demonstrate impressive zero-shot generalization but frequently degrade under diverse real-world illumination, particularly for instance segmentation. In this work, we address this limitation by developing Lighting Convolutional-Attention (LCA), an adapter module that enhances segmentation robustness without fine-tuning the heavy backbone.

Extensive experimental results demonstrate that our approach successfully bridges the domain gap, delivering superior lighting-robust segmentation. p. 0
Where the paper says it

To evaluate and support this architecture, we conduct a comprehensive empirical study across multiple existing benchmarks and present a novel Unity-based synthetic dataset specifically designed to accurately replicate complex real-world lighting conditions. Extensive experimental results demonstrate that our approach successfully bridges the domain gap, delivering superior lighting-robust segmentation.

Pruning redundant reasoning tokens

The authors propose Pairwise Lighting Augmentation Pipeline with Lighting Convolutional Attention (PLAP-LCA), a framework that equips the Segment Anything Model (SAM) with robust lighting invariance without requiring computationally prohibitive fine-tuning of its heavy backbone. At the core of our approach is the LCA module, a specialized adapter injected into SAM’s frozen encoder. p. 1

Where the paper says it

To this end, we propose Pairwise Lighting Augmentation Pipeline with Lighting Convolutional Attention (PLAP-LCA), a framework that equips the Segment Anything Model (SAM) with robust lighting invariance without requiring computationally prohibitive fine-tuning of its heavy backbone. At the core of our approach is the LCA module, a specialized adapter injected into SAM’s frozen encoder.

p. 1
Where the paper says it

To this end, we propose Pairwise Lighting Augmentation Pipeline with Lighting Convolutional Attention (PLAP-LCA), a framework that equips the Segment Anything Model (SAM) with robust lighting invariance without requiring computationally prohibitive fine-tuning of its heavy backbone. At the core of our approach is the LCA module, a specialized adapter injected into SAM’s frozen encoder.

This module employs a dual-branch architecture that explicitly separates standard spatial features from illumination-sensitive structural edge information, leveraging Laplacian-filtered contrast maps. p. 1
Where the paper says it

At the core of our approach is the LCA module, a specialized adapter injected into SAM’s frozen encoder. LCA employs a dual-branch architecture that explicitly separates standard spatial features from illumination-sensitive structural edge information, leveraging Laplacian-filtered contrast maps.

How the training signal is built

The framework utilizes a curriculum-based pairwise training strategy and a novel feature consistency loss to optimize the network. This strategy involves two main components:

  1. Supervised loss: It uses a standard combination of binary cross-entropy (BCE) and Dice loss to train against ground-truth masks, blending both clean and variant streams with a weight of λs = 0. p. 8

    Where the paper says it

    For each predicted mask pk and its corresponding ground truth mk, we use a standard combination of binary cross-entropy (BCE) and Dice loss (induced by Dice Similarity Coefficient): Lseg(p, m) = BCE(p, m) + 1 − Dice(p, m). Both streams contribute to the supervised signal through a weighted blend: Lsup = λs · Lseg(pclean, m) + (1 − λs) · Lseg(pvar, m), where λs = 0.

    5 by default. p. 8
    Where the paper says it

    2 Integration into SAM SAM’s ViT-B encoder consists of 12 transformer blocks. We keep all of them frozen and attach an LCA module to each of the last N blocks (we use N=2 by default).

  2. Consistency loss: This enforces illumination invariance through an L1 penalty between the two streams’ sigmoid-activated outputs, defined as Lcons = σ(pclean) − σ(pvar)1. p. 9

    Where the paper says it

    Dataset Classes Train Val Domain COCO [18] 80 118,287 5,000 General objects VOC 2012 [9] 20 1,464 1,449 General objects Cityscapes [5] 8 2,975 500 Urban driving Unity (Ours) 481 1418 251 Synthetic the consistency loss tells it to predict the same thing regardless of lighting. We enforce this with a simple L1 penalty between the two streams’ sigmoid-activated outputs: Lcons = ∥σ(pclean) − σ(pvar)∥1 .

Where the gains come from

The LCA module is designed to address illumination changes across three distinct dimensions: globally, locally, and structurally. It employs three parallel attention gates—Channel Attention Gate (Gch), Spatial Attention Gate (Gsp), and Contrast Attention Gate (Gct)—to process the input features. p. 5

Where the paper says it

R B×C×H×W (B: batch size, C: number of channels, H: height of feature map, W: width of the feature map): a channel attention gate Gch(R B×C×H×W → R B×C×1×1 ) that recalibrate feature channels to counteract global illumination bias, a spatial attention gate Gsp(R B×C×H×W → R B×1×H×W ) that identifies locally affected regions, and a contrast attention gate Gct(R B×C×H×W → R B×C×1×1 ) based on a Laplacian edge detector that reinforces degraded structural boundaries. The three gates are fused together and projected through a residual path, yielding the complete module: \label {eq:lca_full} \Phi (\mathbf {x}) = \text {PW}\!

The Contrast Attention Gate specifically detects lighting edges by projecting the feature map to a single-channel intensity map and applying a fixed Laplacian kernel. p. 7
Where the paper says it

The projection learns which combination of feature channels best captures lighting-relevant intensity variation. Next, we apply a fixed Laplacian kernel [17] to detect edges in this intensity map: \label {eq:ct_lap} \mathbf {e} = \mathbf {g} * \mathbf {K}_{\text {lap}}, \quad \mathbf {K}_{\text {lap}} = \begin {bmatrix} 0 & 1 & 0 \\ 1 & -4 & 1 \\ 0 & 1 & 0 \end {bmatrix} (3) The Laplacian approximates the second spatial derivative ∇2g, responding strongly wherever intensity changes rapidly.

Robustness Analysis

Empirical results show that LCA improves mean IoU from 0.235 to 0.380 (+61.7%) over the SAM baseline on Cityscapes under lighting variant conditions. p. 9

Where the paper says it

2 Comparative Study Table 2 compares segmentation performance (mIoU) across four datasets under clean (C) and lighting-variant (V) conditions. SAM-0 provides a moderate baseline but shows slight degradation under lighting changes.

Furthermore, LCA recovers a mask (IoU = 0. p. 2
Where the paper says it

Abbreviated paper title 3 Clean Image Variant Image img1 (B, 3, H, W) img2 (B, 3, H, W) SAM Preprocess Resize & Normalization SAM Image Encoder (Frozen) ViT-B + LCA Modules (Ours) Clean Embedding Prompt Encoder Variant Embedding Prompt Sampling Prompt Embedding Mask Decoder (Cross-attention Head) Clean Mask Variant Mask Training Loss Segmentation + Consistency Fig. 1: PLAP-LCA.

585) around a small, low-contrast target in challenging cases where the baseline achieves near-zero IoU (0. p. 7
Where the paper says it

The three gates are combined via element-wise multiplication: \label {eq:fusion} \mathbf {x}_{\text {att}} = \mathbf {x} \odot \underbrace {\mathbf {G}_{\text {ch}}(\mathbf {x})}_{(B,C,1,1)} \odot \underbrace {\mathbf {G}_{\text {sp}}(\mathbf {x})}_{(B,1,H,W)} \odot \underbrace {\mathbf {G}_{\text {ct}}(\mathbf {x})}_{(B,1,H,W)} (5) Broadcasting handles the shape mismatch naturally: the channel gate scales features globally per channel, while the spatial and contrast gates modulate each location uniformly across channels. Multiplication means that if any gate outputs a near-zero value, the corresponding feature is suppressed, regardless of how strongly the other gates activate.

007). The depthwise-separable residual projection is key to the improvement, as it captures per-feature patterns like lighting gradients and penumbra structures. p. 7
Where the paper says it

The depthwise stage applies an independent 3×3 filter to each channel: h = DW(xatt), DW = Conv2d(C, C, 3×3, groups=C). This gives every channel its own local spatial kernel, enabling the module to capture per-feature patterns like lighting gradients and penumbra structures, which cannot be achieved by using a purely pointwise (1×1) projection.

Limitations

Key limitations of PLAP-LCA include extreme lighting like complete darkness with point lights remains untested, the LCA module adds computational cost that may hinder edge deployment, and training requires paired clean-variant images. Conflict sampling relies on manually defined operation groups. p. 3

Where the paper says it

We encode these as conflict groups: \mathcal {C} = \big \{\{\text {warm}, \text {cool}\},\;\{\text {exposure}, \text {brightness}\},\;\{\text {haze}, \text {contrast}\}\big \} (1) Once an operation is selected, everything else in its conflict group is removed from the candidate pool for that image. In the sampling stage, this helps ensure variants remain within the manifold of realistic lighting conditions.

The paper also focuses only on instance segmentation; video and panoptic extensions remain unexplored. p. 2
Where the paper says it

For instance, SAM-Adapter [4] incorporates task-specific tuning, while HQ-SAM [15] introduces an additional token to enhance mask boundary details. More recently, the community has expanded SAM’s capabilities; EfficientSAM [30] drastically reduces computational overhead via masked image pretraining, PerSAM [31] enables training-free, one-shot personalization for specific visual concepts, and SAM 2 [20] extends the foundation model into real-time video segmentation using a streaming memory architecture.

Conclusion and Broader Impact

This work enables safer autonomous driving at night, reliable medical imaging under variable clinical lighting, and robust agricultural monitoring across day-night cycles—all without expensive model retraining or costly real-world data collection. The lightweight adapter design makes illumination robustness accessible for resource-constrained applications. p. 2

Where the paper says it

Our work complements these efforts by addressing the orthogonal challenge of illumination invariance, introducing a lightweight adapter that secures lighting resilience without updating SAM’s pre-trained weights. Robustness to Environmental Conditions.

The LCA module provides substantial lighting invariance through its attention mechanism. p. 1
Where the paper says it

To this end, we propose Pairwise Lighting Augmentation Pipeline with Lighting Convolutional Attention (PLAP-LCA), a framework that equips the Segment Anything Model (SAM) with robust lighting invariance without requiring computationally prohibitive fine-tuning of its heavy backbone. At the core of our approach is the LCA module, a specialized adapter injected into SAM’s frozen encoder.

The zero-initialization ensures stable training, and ablations confirm each attention gate’s complementary contribution. p. 8
Where the paper says it

The network thus starts training as unmodified SAM. Each block then independently learns how much adaptation it needs: blocks processing more lighting-sensitive features tend to develop larger gate values over the course of training, while blocks that already produce stable features keep their gates small.

The LCA+Dec approach delivers the best overall performance on several datasets, demonstrating that combining lighting-aware feature adaptation with decoder tuning improves both segmentation accuracy and lighting invariance. p. 1
Where the paper says it

To this end, we propose Pairwise Lighting Augmentation Pipeline with Lighting Convolutional Attention (PLAP-LCA), a framework that equips the Segment Anything Model (SAM) with robust lighting invariance without requiring computationally prohibitive fine-tuning of its heavy backbone. At the core of our approach is the LCA module, a specialized adapter injected into SAM’s frozen encoder.

The total trainable parameter count for LCA+Decoder Variant is 6. p. 8
Where the paper says it

Each block then independently learns how much adaptation it needs: blocks processing more lighting-sensitive features tend to develop larger gate values over the course of training, while blocks that already produce stable features keep their gates small. The trainable parameter groups in our framework are: (i) the LCA modules attached to the patched blocks, (ii) the per-block gate scalars γℓ, and (iii) SAM’s mask decoder.

4M, which is a 2.6% overhead compared to the SAM ViT-B baseline. p. 2
Where the paper says it

For instance, SAM-Adapter [4] incorporates task-specific tuning, while HQ-SAM [15] introduces an additional token to enhance mask boundary details. More recently, the community has expanded SAM’s capabilities; EfficientSAM [30] drastically reduces computational overhead via masked image pretraining, PerSAM [31] enables training-free, one-shot personalization for specific visual concepts, and SAM 2 [20] extends the foundation model into real-time video segmentation using a streaming memory architecture.

The performance remains stable across a wide range of consistency loss weights, with less than 0. p. 1
Where the paper says it

We propose a pairwise training strategy that optimizes the network using standard segmentation supervision alongside a targeted feature consistency loss, penalizing representational discrepancies across lighting variations. This strategy is evaluated on both real-world public benchmarks and a new synthetic dataset generated from a Unity environment, covering a wide range of physically accurate illumination.

15% mIoU variation between the best and worst settings. p. 0
Where the paper says it

This creates a critical gap in realworld applications such as nighttime driving, indoor low-light scenes, overexposed outdoor settings, and mixed artificial and natural lighting conditions. In safety-critical applications such as autonomous driving [22], robotic manipulation, and medical imaging [1], lighting variation is the norm rather than the exception.

--- Page 1 ---

Lighting-aware Unified Model for Instance Segmentation Qisai Liu, Alloy Das, Zhanhong Jiang, Joshua R. Waite, Aditya Balu, Adarsh Krishnamurthy, and Soumik Sarkar Iowa State University, Ames, IA 50011, USA

Abstract. Foundation models like the Segment Anything Model (SAM) demonstrate impressive zero-shot generalization but frequently degrade under diverse real-world illumination, particularly for instance segmentation. In this work, we address this limitation by developing Lighting Convolutional-Attention (LCA), an adapter module that enhances segmentation robustness without fine-tuning the heavy backbone. p. 0

Where the paper says it

Foundation models like the Segment Anything Model (SAM) demonstrate impressive zero-shot generalization but frequently degrade under diverse real-world illumination, particularly for instance segmentation. In this work, we address this limitation by developing Lighting Convolutional-Attention (LCA), an adapter module that enhances segmentation robustness without fine-tuning the heavy backbone.

LCA employs a dual-branch architecture to process RGB features alongside contrast maps, enabling physically motivated sensitivity to structural changes rather than illumination artifacts. p. 0
Where the paper says it

In this work, we address this limitation by developing Lighting Convolutional-Attention (LCA), an adapter module that enhances segmentation robustness without fine-tuning the heavy backbone. LCA employs a dual-branch architecture to process RGB features alongside contrast maps, enabling physically motivated sensitivity to structural changes rather than illumination artifacts.

We optimize LCA through a pairwise training strategy, introducing a targeted loss term that explicitly penalizes discrepancies between clean images and their corresponding illumination variants. p. 0
Where the paper says it

LCA employs a dual-branch architecture to process RGB features alongside contrast maps, enabling physically motivated sensitivity to structural changes rather than illumination artifacts. We optimize LCA through a pairwise training strategy, introducing a targeted loss term that explicitly penalizes discrepancies between clean images and their corresponding illumination variants.

To evaluate and support this architecture, we conduct a comprehensive empirical study across multiple existing benchmarks and present a novel Unity-based synthetic dataset specifically designed to accurately replicate complex real-world lighting conditions. p. 0
Where the paper says it

We optimize LCA through a pairwise training strategy, introducing a targeted loss term that explicitly penalizes discrepancies between clean images and their corresponding illumination variants. To evaluate and support this architecture, we conduct a comprehensive empirical study across multiple existing benchmarks and present a novel Unity-based synthetic dataset specifically designed to accurately replicate complex real-world lighting conditions.

Keywords: Instance segmentation · Lighting self-attention · Curriculumbased pairwise training. p. 0

Where the paper says it

Extensive experimental results demonstrate that our approach successfully bridges the domain gap, delivering superior lighting-robust segmentation. Keywords: Instance segmentation · Lighting self-attention · Curriculumbased pairwise training 1 Introduction Instance segmentation has achieved remarkable progress through foundation models such as the Segment Anything Model (SAM) [16], which demonstrates impressive zero-shot generalization across diverse visual domains.

--- Page 2 ---

Q. Liu, A. Das et al. 1 Introduction Instance segmentation has achieved remarkable progress through foundation models such as the Segment Anything Model (SAM) [16], which demonstrates impressive zero-shot generalization across diverse visual domains. p. 0

Where the paper says it

Extensive experimental results demonstrate that our approach successfully bridges the domain gap, delivering superior lighting-robust segmentation. Keywords: Instance segmentation · Lighting self-attention · Curriculumbased pairwise training 1 Introduction Instance segmentation has achieved remarkable progress through foundation models such as the Segment Anything Model (SAM) [16], which demonstrates impressive zero-shot generalization across diverse visual domains.

However, because these models are predominantly trained on well-lit and high-quality imagery, they frequently suffer significant performance degradation when deployed in environments with adverse illumination. p. 0
Where the paper says it

Keywords: Instance segmentation · Lighting self-attention · Curriculumbased pairwise training 1 Introduction Instance segmentation has achieved remarkable progress through foundation models such as the Segment Anything Model (SAM) [16], which demonstrates impressive zero-shot generalization across diverse visual domains. However, because these models are predominantly trained on well-lit and high-quality imagery, they frequently suffer significant performance degradation when deployed in environments with adverse illumination.

This creates a critical gap in realworld applications such as nighttime driving, indoor low-light scenes, overexposed outdoor settings, and mixed artificial and natural lighting conditions. p. 0
Where the paper says it

However, because these models are predominantly trained on well-lit and high-quality imagery, they frequently suffer significant performance degradation when deployed in environments with adverse illumination. This creates a critical gap in realworld applications such as nighttime driving, indoor low-light scenes, overexposed outdoor settings, and mixed artificial and natural lighting conditions.

In safety-critical applications such as autonomous driving [22], robotic manipulation, and medical imaging [1], lighting variation is the norm rather than the exception. p. 0

Where the paper says it

This creates a critical gap in realworld applications such as nighttime driving, indoor low-light scenes, overexposed outdoor settings, and mixed artificial and natural lighting conditions. In safety-critical applications such as autonomous driving [22], robotic manipulation, and medical imaging [1], lighting variation is the norm rather than the exception.

Similarly, in agricultural applications, plant segmentation in outdoor scenes is subject to drastic illumination shifts from morning to night, leading to significant model degradation [23]. p. 1
Where the paper says it

scenes is subject to drastic illumination shifts from morning to night, leading to significant model degradation [23]. A segmentation model that performs well under controlled laboratory conditions but degrades under a cloudy sky or in a dimly lit warehouse is fundamentally unreliable.

We argue that addressing lighting robustness requires intervention at three complementary levels: (1) the data level, where training images must systematically span the complex space of realistic lighting conditions; (2) the architectural level, where the model must possess explicit mechanisms for reasoning about structural integrity independent of illumination; and (3) the training level, where the learning objective must explicitly force the extraction of lighting-invariant representations. p. 1

Where the paper says it

Yet, the dominant paradigm in segmentation research treats lighting variation as a secondary concern, addressed primarily through generic data augmentation [24] or domain adaptation [27], rather than as a first-class architectural and training problem. We argue that addressing lighting robustness requires intervention at three complementary levels: (1) the data level, where training images must systematically span the complex space of realistic lighting conditions; (2) the architectural level, where the model must possess explicit mechanisms for reasoning about structural integrity independent of illumination; and (3) the training level, where the learning objective must explicitly force the extraction of lighting-invariant representations.

Contributions. p. 1

Where the paper says it

We argue that addressing lighting robustness requires intervention at three complementary levels: (1) the data level, where training images must systematically span the complex space of realistic lighting conditions; (2) the architectural level, where the model must possess explicit mechanisms for reasoning about structural integrity independent of illumination; and (3) the training level, where the learning objective must explicitly force the extraction of lighting-invariant representations. Contributions.

To this end, we propose Pairwise Lighting Augmentation Pipeline with Lighting Convolutional Attention (PLAP-LCA), a framework that equips the Segment Anything Model (SAM) with robust lighting invariance without requiring computationally prohibitive fine-tuning of its heavy backbone. p. 1
Where the paper says it

Contributions. To this end, we propose Pairwise Lighting Augmentation Pipeline with Lighting Convolutional Attention (PLAP-LCA), a framework that equips the Segment Anything Model (SAM) with robust lighting invariance without requiring computationally prohibitive fine-tuning of its heavy backbone.

At the core of our approach is the LCA module, a specialized adapter injected into SAM’s frozen encoder. LCA employs a dual-branch architecture that explicitly separates standard spatial features from illumination-sensitive structural edge information, leveraging Laplacian-filtered contrast maps. p. 1
Where the paper says it

At the core of our approach is the LCA module, a specialized adapter injected into SAM’s frozen encoder. LCA employs a dual-branch architecture that explicitly separates standard spatial features from illumination-sensitive structural edge information, leveraging Laplacian-filtered contrast maps.

We propose a pairwise training strategy that optimizes the network using standard segmentation supervision alongside a targeted feature consistency loss, penalizing representational discrepancies across lighting variations. p. 1
Where the paper says it

LCA employs a dual-branch architecture that explicitly separates standard spatial features from illumination-sensitive structural edge information, leveraging Laplacian-filtered contrast maps. We propose a pairwise training strategy that optimizes the network using standard segmentation supervision alongside a targeted feature consistency loss, penalizing representational discrepancies across lighting variations.

Related Work

Segmentation Models. The Segment Anything Model (SAM) [16] represents a paradigm shift in image segmentation, leveraging a dataset of over 1 billion masks to achieve strong zero-shot generalization through a promptable architecture. p. 1

Where the paper says it

Segmentation Models. The Segment Anything Model (SAM) [16] represents a paradigm shift in image segmentation, leveraging a dataset of over 1 billion

Subsequent work has explored SAM adaptation for specialized domains, but existing adaptations primarily optimize for computational efficiency or boundary precision under standard conditions. p. 2
Where the paper says it

More recently, the community has expanded SAM’s capabilities; EfficientSAM [30] drastically reduces computational overhead via masked image pretraining, PerSAM [31] enables training-free, one-shot personalization for specific visual concepts, and SAM 2 [20] extends the foundation model into real-time video segmentation using a streaming memory architecture. However, despite these rapid architectural evolutions, existing adaptations primarily optimize for computational efficiency, semantic tracking, or boundary precision under standard conditions.

Dataset Generation

A main contribution of our approach is that segmentation models can be trained to be lighting-invariant, with structured pairwise training on clean and lighting-variant images that share identical ground-truth masks. We employ two complementary strategies for generating realistic lighting variants:

  1. Image-space augmentation by the Pairwise Lighting Augmentation Pipeline (PLAP), which applies physically motivated operations like exposure, directional shadow, and color temperature to existing datasets. p. 3

    Where the paper says it

    The pipeline is defined by three design elements: physically motivated operations, conflict awareness sampling, and structured severity levels. PLAP implements multiple (in this work we select nine to cover generic scenarios) lighting operations, each corresponded to physical models of image formation or sensor response: exposure, directional shadow, color temperature (warm/cool), vignetting, contrast, Gamma correction, brightness, film grain, and atmospheric haze.

  2. Physically-rendered synthetic data generated within the Unity engine, which provides complex illumination variations with pixel-level perfect annotations. p. 3

    Where the paper says it

    This requires robust methods for generating realistic lighting variants. We employ two complementary strategies: (1) image-space augmentation by the Pairwise Lighting Augmentation Pipeline (PLAP), which is applicable to any existing dataset, and (2) physically rendered synthetic data generated within the Unity engine, which provides complex illumination variations with pixel-level perfect annotations.

Methodology

The framework consists of three components: (1) a pairwise lighting augmentation pipeline for generating training pairs, PLAP, which was detailed in Sec. 3. (2) an LCA module for illumination awareness feature processing. p. 4

Where the paper says it

4 Methodology Our framework consists of three components: (1) a pairwise lighting augmentation pipeline for generating training pairs, PLAP, which was detailed in Sec. 3 (2) an LCA module for illumination awareness feature processing, and (3) a pairwise training strategy with driven by a novel consistency loss function.

(3) a pairwise training strategy driven by a novel consistency loss function. p. 4
Where the paper says it

4 Methodology Our framework consists of three components: (1) a pairwise lighting augmentation pipeline for generating training pairs, PLAP, which was detailed in Sec. 3 (2) an LCA module for illumination awareness feature processing, and (3) a pairwise training strategy with driven by a novel consistency loss function.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Lighting-invariant feature extraction: The system can process images under diverse real-world illumination, such as nighttime driving or indoor low-light scenes, because it uses a dual-branch architecture to process RGB features alongside contrast maps and Laplacian-filtered maps to obtain physically motivated sensitivity to structural changes rather than illumination artifacts.

  2. Robust training strategy: The model can be trained with a targeted loss term that explicitly penalizes discrepancies between clean images and their corresponding illumination variants, which helps the model learn lighting-invariant representations.

  3. Adaptive attention mechanism: The system can selectively focus on relevant structural information by employing three parallel attention gates—channel, spatial, and contrast—to determine where to attend spatially and which feature channels matter.

  4. Zero-shot generalization enhancement: By injecting the Lighting Convolutional Attention (LCA) module into a frozen backbone, the system can achieve superior lighting-robust segmentation even when deployed in environments it was not specifically fine-tuned for.

  5. Synthetic data generation: The framework can be supported by a novel Unity-based synthetic dataset that replicates complex real-world lighting conditions through a physically rendered engine, allowing for training on data where ground truth instance masks are captured natively by the Unity Perception package.

Abstract

Foundation models like the Segment Anything Model (SAM) demonstrate impressive zero-shot generalization but frequently degrade under diverse real-world illumination, particularly for instance segmentation. In this work, we address this limitation by developing Lighting Convolutional-Attention (LCA), an adapter module that enhances segmentation robustness without fine-tuning the heavy backbone. LCA employs a dual-branch architecture to process RGB features alongside contrast maps, enabling physically motivated sensitivity to structural changes rather than illumination artifacts. We optimize LCA through a pairwise training strategy, introducing a targeted loss term that explicitly penalizes discrepancies between clean images and their corresponding illumination variants. To evaluate and support this architecture, we conduct a comprehensive empirical study across multiple existing benchmarks and present a novel Unity-based synthetic dataset specifically designed to accurately replicate complex real-world lighting conditions. Extensive experimental results demonstrate that our approach successfully bridges the domain gap, delivering superior lighting-robust segmentation.

Sources

Related papers