VibeEdit: Image Editing with Canvas Instructions

The gist The VibeEdit system introduces an image editing interface where users place spatial marks and optional short notes directly onto an image to specify editing intent, allowing it to perform object addition, removal, replacement, attribute modification, and movement without requiring a separate text prompt.

Introduction

The paper introduces a new image editing interface that lets users place spatial marks and optional short notes directly on the image (Figure 1) Together these annotations form a canvas instruction that specifies where to edit and what to change. Circles identify objects or regions, scribbles indicate removal, and arrows specify movement. Short notes placed near the marks describe the desired content or attributes. To support this interface, VibeEdit is developed as an image editor that follows canvas instructions to perform object addition, removal, replacement, attribute modification, and movement without a separate text prompt.

Training Methodology

VibeEdit is trained using paired examples connecting canvas instructions to the intended edits. The training involves constructing a corpus of 1.55 million source–target edit pairs by selecting editable objects and generating edited targets with existing models. During training, spatial marks and short notes are rendered, sampling either editing direction where applicable. The model adapts Qwen-Image-Edit with layer-decoupled conditioning, where the vision–language encoder interprets the annotated image, while the generative pathway separately encodes the source image and canvas annotations. Because most pixels remain unchanged in a localized edit, region-weighted supervised fine-tuning is used to emphasize the edit region and annotation removal while retaining supervision over the full image. Finally, rubric-guided reinforcement learning refines the editor using rewards for edit completion, local edit quality, and preservation of unedited regions.

Training Data Construction

The supervised learning corpus is constructed by selecting editable objects and generating edited targets with existing models. Object selection and prompt synthesis involve Florence-2 generating openvocabulary object proposals, which are indexed by GPT-5.6-sol (OpenAI, 2026b) to select a recognizable object. SAM 3 produces an object mask M from the selected box. For attribute modification, GPT-5.6-sol selects one visible attribute and specifies a distinct, verifiable target value. Object replacement uses FLUX.1 Fill (Black Forest Labs, 2024) to generate the replacement described by the edit prompt within an expanded version of M. Object movement involves GPT-5.6-sol selecting a movable object and a clearly separated destination on a suitable, unoccupied support surface, followed by SAM 3 to obtain the destination mask Mtgt. The final corpus contains 1,554,062 stored image pairs.

Model Architecture and Conditioning

The training pipeline utilizes layer-decoupled conditioning where the vision–language encoder processes the annotated image A to produce semantic tokens h = EVL(A). The generative pathway separately encodes the source image I and rasterized instruction C¯ to obtain zI = EVAE(I) and zC = EVAE(C¯). The DiT receives noisy target tokens, source tokens zI, canvas tokens zC, and semantic tokens h. The model is refined using rank-128 LoRA (Hu et al., 2021) adapters on the DiT attention projections and modulation layers.

Evaluation and Results

VibeEdit is evaluated on an independently constructed, human-curated benchmark of 419 cases emphasizing target selection among similar objects. VibeEdit achieves an overall VLM rubric score of 79.9 and an outside-region PSNR of 32.8 dB, compared with 67.4 and 24.0 dB for FireRed, the highest-scoring text-instructed baseline in our evaluation. The final VibeEdit configuration achieves the highest rubric score on all five edit types and the highest overall PSNR, reaching 79.9 and 32.8 dB, respectively.

Ablation Studies

In ablation studies, encoding only the source image I leaves the canvas instruction represented solely by semantic tokens and yields a rubric score of 24.7. Combining weighting with dilation raises the rubric score to 67.8. The combination of all three rubric groups—edit success, outside preservation, and local edit quality—achieves the best results, with a rubric score of 79.9 and PSNR of 32.8 dB.

Conclusion

We introduced a new image editing interface in which spatial marks and optional short notes form canvas instructions, specifying where to edit and what to change. We developed VibeEdit to support five editing operations through this interface without a separate text prompt. Using 1.55 million source–target edit pairs, we trained VibeEdit with layer-decoupled conditioning, region-weighted supervised learning, and rubric-guided RL. On an independent, human-curated 419-case benchmark emphasizing target selection among similar objects, VibeEdit outperforms the evaluated baselines in both overall VLM rubric score and outside-region PSNR.

AI Use Statement

The authors developed the research ideas and methodology without generative AI assistance. We used GPT models for object selection and edit-description generation, and generative image models to synthesize training pairs, as described in Section 3. GPT models also provided rubric-based rewards during reinforcement learning and evaluated editing outputs. We also used language models to improve the clarity and readability of the manuscript.

Ethics Statement

Our data include images from publicly available datasets and AI-generated images.

Reproducibility Statement

To ensure reproducibility, Sections 3, 4, and 5.1 describe data construction, the model and training objectives, and the training, inference, and evaluation settings. The appendix provides additional data details, the RL policy update, data-selection rules, and evaluation prompts and rubrics.

References

Shuai Bai et al. Qwen2.5-vl technical report 2025 https://arxiv.org/abs/2502.13923 https://arxiv.org/abs/2610.12229

Kevin Black et al Training diffusion models with reinforcement learning In International Conference on Learning Representations volume 4965–4987 2024

Black Forest Labs Introducing FLUX.1 tools https://bfl.ai/blog/24-11-21-tools 2024

Black Forest Labs FLUX.2 [klein] https://bfl.ai/models/flux-2-klein 2026

Tim Brooks et al Instructpix2pix Learning to follow image editing instructions In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition pp. 18392–18402 IEEE, 2023

Huanqia Cai et al Z-image An efficient image generation foundation model with single-stream diffusion transformer arXiv preprint arXiv:2511.22699 2025

Siyu Cao et al Hunyuanimage 3.0 technical report arXiv preprint arXiv:2509.23951 2025

Nicolas Carion et al Sam 3 Segment anything with concepts In International conference on learning representations volume 138846–138923 2026

Dong Chen et al Lens Rethinking training efficiency for foundational text-to-image models arXiv preprint arXiv:2605.21573 2026

Xuelu Feng et al Rubicrl Simple generalizable rewards for text-to-image generation arXiv preprint arXiv:2511.20651 2025

Google Gemini 2.5 Flash Image Nano Banana 2025 https://ai.google.dev/gemini-api/docs/models/gemini-2.5-flash-image

Edward J Hu et al Lora Low-rank adaptation of large language models arXiv preprint arXiv:2106.09685 2021

Delong Liu et al Inter-edit First benchmark for interactive instruction-based image editing In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition pp. 37290–37300 2026a

Jie Liu et al Flow-grpo Training flow matching models via online rl In Advances in Neural Information Processing Systems volume 38 2025a doi: 10.52202/085713-1362

Shiyu Liu et al Step1x-edit A practical framework for general image editing arXiv preprint arXiv:2504.

1. The VibeEdit interface allows users to place spatial marks and optional short notes directly on the image, forming a canvas instruction that specifies where to edit and what to change. This enables direct spatial grounding for editing operations, complementing conventional text prompting.

2. VibeEdit supports diverse editing operations including object addition, removal, replacement, attribute modification, and movement by following canvas instructions without needing a separate text prompt.

3. The training methodology adapts Qwen-Image-Edit with layer-decoupled conditioning that separately encodes source images and canvas instructions for image editing, providing spatial guidance while retaining a clean source reference.

4. Region-weighted supervised fine-tuning emphasizes the edit region by defining a loss: LSFT = Et,ϵ q1 + λMf ⊙ (vθ(zt; h, zI, zC, t) − v∗) where Mf is dilated to extend the upweighted region around object boundaries and annotations.

5. Rubric-guided reinforcement learning refines editing behavior using rewards that assess edit completion, local edit quality, and preservation of unedited regions, specifically penalizing failures like vlm no global style drift or vlm instruction no extra edits.

6. The system can perform high-fidelity object selection and prompt synthesis by utilizing components like Florence-2 for open-vocabulary proposals and GPT-5.6-sol to select objects, names, and generate edit prompts.

7. The model demonstrates robust performance on similar objects because the human benchmark emphasizes scenes with similar objects to test whether the editor applies the requested change to the intended instance.

8. Ablation studies confirm that separately encoding I and C¯ further improves the rubric score to 67.8 and PSNR from 24.2 to 27.0 dB, validating the strategy of encoding both source image and canvas instructions separately for generation.