Domain-Grounded Candidate Selection for Agentic Image Editing: A Shadow Removal Case

Summary

This paper, Domain-Grounded Candidate Selection for Agentic Image Editing: A Shadow Removal Case by Shilin Hu, Jingyi Xu, Dimitris Samaras, and Hieu Le, investigates whether commercial vision-language models eliminate the need for classic, physics-informed low-level vision, using shadow removal as a test case. The authors note that "Commercial vision-language models are reshaping computer vision, with visual priors broad enough to rival task-specific systems. This raises a natural question: do they reduce the need for classic, physics-informed low-level vision? Shadow removal is a particularly suitable problem to study this because shadow appearance depends on scene geometry, lighting, materials, and occluders, and paired shadow and shadow-free training data are difficult to collect at scale."

The authors find that a commercial generative editor, used directly, can produce clean shadow-free edits that preserve surface texture and local appearance better than prior shadow-removal methods. However, "the same editor has no explicit notion of how a shadow forms, so a single query can fail in a specific way: it may remove a shadow by regenerating scene content, misread reduced illumination as material or geometry, or hallucinate new objects and textures into the target region, producing edits that look convincing while being physically wrong."

To address this, they propose an agentic candidate-selection pipeline: "the editor generates a guided probe, an evaluator screens for major failures, retries when needed, samples multiple candidates, filters them, and selects a final result balancing shadow removal against scene preservation. Crucially, they ground this process in shadow-formation physics: Grounding this process in shadow-formation physics makes it more reliable: prompting the generator and evaluator to treat shadows as illumination effects caused by light occlusion, not material or object structure, measurably improves quality and consistency."

Method. Given an input image I and shadow mask M, the goal is to produce a shadow-removed image Iˆ while preserving scene layout, object structure, and non-shadow appearance. The system uses a generative image editor as a stochastic candidate generator and a multimodal evaluator as the agentic controller. The pipeline begins with "a guided probe: a single mask-guided edit generated from the input image, shadow mask, and shadow removal instruction. A major-failure evaluator rejects severe errors, including preserving the shadow as an object-like dark structure or hallucinating semantic content in the target region. Then, the probe outcome determines candidate generation. If the probe passes, we keep it and sample two additional guided candidates; otherwise, we generate three retry candidates using evaluator feedback. All candidates are filtered by the same major-failure evaluator. When multiple candidates pass, a minor-quality selector chooses the final output by balancing shadow removal with scene preservation; when one candidate passes, it is selected directly; when none pass, a fallback selector chooses the least-bad candidate."

The authors compare two prompt settings under the same pipeline: "The generic setting uses direct shadow removal and preservation instructions. The physics-oriented setting adds simple shadow-formation grounding, prompting the generator and evaluator to treat shadows as illumination effects caused by light occlusion rather than material or object structure. This encourages removal of reduced illumination while preserving scene geometry and material appearance."

Implementation and evaluation. The system uses "GPT-image-2 for candidate generation and GPT-5-mini from the GPT-5 family for multimodal evaluation and selection, with all calls made through the OpenAI API. Each full benchmark run processes the SRR benchmark once, takes roughly 14 hours, and uses up to four image-editing calls per case plus evaluator/selector calls. Evaluation is on the ShadowRemovalRefine (SRR) benchmark, with all benchmark images and masks normalized to 512×512. The benchmark contains 400 images; one image was rejected by the API, leaving 399 evaluated cases. They report Color Distribution Difference (CDD), which measures color-distribution discrepancy across annotated shadow boundaries; lower CDD indicates better shadow removal and boundary consistency, and compare against SID, ShadowFormer, ShadowDiffusion, Inpaint4Shadow, and SRR. Because image generation is stochastic, we run each agentic variant twice and report averaged statistics across the two full runs."

Quantitative results. "Ours-Physics achieves the lowest mean CDD among all evaluated methods, reaching 0.0075 and reducing CDD by at least 47.0% compared with the strongest prior method, SRR. Ours-Generic also improves over prior methods, indicating that combining a generative image editor with candidate selection is effective, while shadow-formation grounding further improves both mean CDD and standard deviation." Specifically, Ours-Physics attains a mean CDD of 7.45×10−3 (std 20.65), versus 10.33 for Ours-Generic and 14.06 for SRR.

Qualitative results. "Existing methods often attenuate the shadow but leave visible boundary traces, residual dark regions, or washed-out surface appearance. Our method more consistently removes the dominant shadow region and suppresses shadow-edge artifacts, producing outputs that better match the surrounding scene appearance."

Ablation analysis. The ablation of pipeline stages shows "the selected output improves over the initial probe, showing that major-failure filtering, retry, and final selection are more reliable than using a single stochastic edit. The physics-oriented setting further improves over the generic setting, suggesting that shadow-formation grounding helps the generator and evaluator prefer candidates with cleaner shadow removal. In the physics setting, CDD improves from 9.19 for the probe to 7.45 for the final selected output; in the generic setting, from 12.07 to 10.33. Retry cases are shown where the major-failure evaluator rejects a failed probe and triggers a cleaner retry candidate, and figures illustrate that the generic pipeline can still leave residual shadows, introduce boundary artifacts, or make task-level mistakes when interpreting dark regions, while physics-oriented prompting produces a stronger candidate set and final selection chooses the candidate that best balances shadow removal with scene consistency."

Limitations. "Although our pipeline improves shadow-boundary CDD, it does not always preserve the entire non-shadow scene. As shown in Tab. 3 and Fig. 7, generative editing can introduce appearance shifts outside the target shadow mask, including changes in color tone, surface smoothness, and fine structure. It can also modify dark geometric regions when they are visually entangled with the shadow. These failures suggest that candidate selection improves reliability but does not fully solve preservation when using general-purpose generative editors." The non-shadow preservation metric (RGB RMSE outside the mask) is higher for their method (25.75 generic, 28.79 physics) than for prior methods (e.g., SID 3.44), indicating a preservation tradeoff.

Conclusion and future work. "We present a domain-grounded candidate-selection pipeline for controllable shadow removal with general-purpose generative image editors. By combining probing, major-failure retry, candidate filtering, final selection, and shadow-formation grounding, the method achieves state-of-the-art CDD on the SRR benchmark without training a task-specific restoration model. The authors conclude that Foundation models, in other words, do not retire classic low-level vision; they change its role, from modeling a phenomenon end-to-end to steering and correcting a much more powerful but physically underconstrained generator. Future work will refine selected outputs with additional learned correction or fine-tuning stages to improve preservation and reliability, and they plan to curate a large-scale real-world shadow removal benchmark with paired shadow and shadow-free examples, covering more complex scenes, materials, illumination conditions, and shadow configurations than existing datasets."

Based on this paper, here are specific improvements for AI image-editing and agentic vision-language systems:

1. Add physics-grounded prompting to generative editors and evaluators.

• Improvement: Before generating an edit, inject domain knowledge: “Shadows are illumination effects caused by light occlusion; do not treat them as material, geometry, or new objects. Remove the reduced illumination while preserving surface texture, material color, object structure, and scene layout.”

• Resulting capability: The system distinguishes shadow removal from content regeneration, reducing hallucinations and avoiding deletion of dark, non-shadow objects.

2. Give the system an explicit “probe → evaluate → retry → select” agentic pipeline.

• Improvement: Instead of using a single generative edit, the system first produces one guided probe edit, a multimodal evaluator checks it for major failures, and if it fails, the system retries with evaluator feedback before sampling multiple candidates.

• Resulting capability: The system can self-correct, catching cases where it wrongly preserves a shadow as a dark object-like structure or hallucinates new content.

3. Add a two-level evaluator: major-failure filtering followed by minor-quality selection.

• Improvement: Use a strict filter to reject semantically wrong outputs, then a second selector that ranks surviving candidates by balancing shadow-removal quality against scene preservation.

• Resulting capability: The system avoids convincing but physically wrong edits and reliably chooses the output that best fits the surrounding non-shadow appearance.

4. Exploit stochasticity deliberately by sampling multiple candidates and selecting, not settling on one sample.

• Improvement: Generate several guided candidates from the same input, filter them, and select the best one.

• Resulting capability: The system becomes more robust, with lower variance and better boundary consistency than any single commercial generative edit.

5. Separate “remove the shadow” from “preserve the scene” as two explicit, competing objectives.

• Improvement: Have the selector score candidates on both dimensions instead of a single overall quality score.

• Resulting capability: The system can make a principled tradeoff, improving shadow-boundary consistency while flagging or avoiding unwanted color-tone, smoothness, and structure changes outside the shadow mask.

6. Add a post-selection preservation-refinement stage.

• Improvement: After picking the best physics-grounded candidate, apply a learned correction or fine-tuning stage that constrains non-shadow regions to match the original input’s color distribution, texture, and geometry.

• Resulting capability: The system would reduce outside-mask RGB RMSE, addressing the paper’s main limitation, and produce edits that preserve the entire non-shadow scene, not just the shadow boundary.

7. Generalize this domain-grounded agentic selection to other image-formation problems.

• Improvement: Use the same probe–retry–filter–select pattern with physics-grounded prompts for reflection removal, dehazing, or illumination correction, where the underlying cause is also a physical light-transport effect.

• Resulting capability: A single agentic framework can handle multiple low-level vision tasks robustly without task-specific restoration networks.

The improved AI system can, specifically: remove soft and hard shadows while preserving textural details; detect and retry its own failures; avoid hallucinating new objects in shadow regions; avoid modifying dark materials that merely resemble shadows; outperform prior shadow-removal methods on boundary color-distribution consistency; and reduce run-to-run variability, while being extendable to enforce strict non-shadow preservation.

Commercial vision-language models are reshaping computer vision, with visual priors broad enough to rival task-specific systems. This raises a natural question: do they reduce the need for classic, physics-informed low-level vision? We study this through shadow removal, a problem shaped by scene geometry, illumination, materials, and occluders, where paired shadow and shadow-free data are hard to collect at scale. We find that a commercial generative editor, used directly, can produce clean shadow-free edits that preserve surface texture and local appearance. However, this comes with a new failure mode: the same editor can regenerate scene content, hallucinate objects, or misread a shadow as material or geometry, producing plausible but physically wrong edits. We address this with an agentic candidate-selection pipeline: the editor generates a guided probe, an evaluator screens for major failures, retries when needed, samples multiple candidates, filters them, and selects a final result balancing shadow removal against scene preservation. Grounding this process in shadow-formation physics makes it more reliable: prompting the generator and evaluator to treat shadows as illumination effects caused by light occlusion, not material or object structure, measurably improves quality and consistency. On the ShadowRemovalRefine benchmark, our physics-oriented pipeline achieves a CDD of 0.0075, reducing CDD by at least 47% over the strongest prior method. These results suggest that commercial vision-language models do not replace classic low-level vision priors; instead, such priors remain useful for constraining and steering physically underconstrained generation.