Imagine Before You Draw: Visual Prompt Engineering for Image Generation

arXiv:2606.04457 · cs.CV · Submitted 2026-06-03 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "Imagine Before You Draw".

Jane: The gist: Visual Prompt Engineering (VPE) improves image generation quality and editing preservation by inserting SigLIP 2 visual tokens as an intermediate semantic plan,

Tom: First, who's behind it and why it matters.

Paper summary: Tom: So we’ve seen how Visual Prompt Engineering uses an intermediate visual plan to help image models connect text and pixels more effectively, and now we’re looking at the paper "Imagine Before You Draw: Visual Prompt Engineering for Image Generation."

Jane: That title really suggests a shift in thinking about image creation, implying that pausing before you draw helps visualize the structure first.

Lu: The authors are tackling that core problem of bridging the gap between text and pixels by inserting this visual plan into the generation process.

Meng: From my side, I’m interested in how they handle training stability because getting a reliable intermediate step isn't always smooth when you’re trying to scale these things up for production.

Tom: Well, what they show here is that by integrating this visual prompting directly into existing internal architectures, you don’t just get a slightly better image; you see real gains in how well the model performs editing tasks.

Jane: Editing preservation, right? That means when someone tells the AI to change just one thing in a picture, it keeps all the other details intact. And these results suggest this method does that much better than what they saw in external setups.

Lu: The difference they show is significant, especially when you look at structure distance metrics and PSNR improvements compared to those separate models they tested alongside their internal ones. It seems like it really helps the model retain that fine detail during refinement.

Meng: So for someone who actually has to deploy this stuff in a production setting, the fact that it boosts editing quality without necessarily adding a huge amount of extra compute is something worth considering.

Tom: Right, it’s about making internal AI models more robust when they have to handle complex instructions and maintain fidelity during iterative changes.

Jane: It really shifts the focus from just generating *a* picture to generating a picture with specific, structured intent that can be manipulated later, which is a big step forward for creative tools.

Lu: And it opens up more avenues for how we condition these models; it’s not just about text input anymore; it’s about injecting rich visual planning data directly into the generation loop.

Meng: So the next thing we should look at is whether this method stays stable when you try to apply it to entirely different types of images or prompts outside of those specific class-conditional tests they ran.

Conclusion: Jane: So, we've been talking about how this paper uses these visual plans to make text and images talk to each other better, and now we’re looking at what they actually named their work: "Imagine Before You Draw: Visual Prompt Engineering for Image Generation."

Tom: That title really suggests a whole new way of thinking about image creation. It sounds like the authors are proposing that you should visualize the structure first before you actually draw or generate anything complex.

Lu: Exactly. The core idea they’re pushing is to solve that gap between what text describes and what a pixel actually looks like by putting this visual plan right into the generation process itself.

Meng: From my side, I’m still thinking about how they handle the training stability on a real scale, because getting that reliable intermediate step isn't always smooth when you try to run these things in production.

Tom: Well, what they show here is that by integrating this visual prompting directly into existing internal architectures—like those diffusion models—you don’t just get a slightly better image; you see real gains when the model performs editing tasks.

Jane: Editing preservation, right? That means when someone tells the AI to change just one thing in a picture, it keeps all those other details exactly where they are supposed to be. And these results suggest this method does that much better than what we’re seeing in external setups.

Lu: The authors are also clear that they looked at different types of tokens, like discrete tokens and continuous vectors, to see how those affect how fast the model converges and what the quality ceiling ends up being.

Meng: They also flagged a limitation, which is that learning those reliable visual plans isn't easy; if you just train the model normally on ground-truth prompts, that plan might not actually be useful for anything real.

Tom: So what we have here is a solid way to improve generation quality and give models better control over their output through this Visual Prompt Engineering method.

Jane: It really shifts the focus from just making *a* picture to making a picture with specific, structured intent that you can manipulate later, which is a big step forward for design tools.

Lu: And it opens up more ways to condition these models—it’s not just about text input anymore; it’s about injecting rich visual planning data directly into the generation loop.

Meng: So the next thing we should look at is whether this method stays stable when you try to apply it to entirely different types of images or prompts outside of those specific class-conditional tests they ran.

Liyu Jia, *, Fengda Zhang, *, †Jiachun Pan, *, Kesen Zhao1 Saining Zhang1 Wang Lin3 Weijia Wu2 Yue Liao2 Aojun Zhou4 Hanwang Zhang1

Nanyang Technological University · National University of Singapore · Zhejiang University · The Chinese University of Hong Kong

cs.CV

Submitted: 2026-06-03

Updated: 2026-10-14

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 83/100

The gist: The gist: Visual Prompt Engineering (VPE) improves image generation quality and editing preservation by inserting SigLIP 2 visual tokens as an intermediate semantic plan, which addresses the modeling

Key concepts

Visual Semantic Tokens (VPT)
These are visual representations, specifically SigLIP 2 tokens extracted from a target image. They act as an intermediate semantic plan that captures the layout and meaning of the original image. Instead of directly mapping text to pixels, the model first learns to generate these meaningful visual prompts.
Visual Prompt Engineering (VPE)
VPE is a technique where the model generates these visual semantic tokens (VPT) autoregressively before generating the final image tokens. This intermediate plan guides the subsequent detail rendering of the image, effectively bridging the gap between text and pixels by providing a structured semantic roadmap.
Information Bottleneck
This occurs in external pipelines where an independent model generates semantic tokens which are then fed to a separate decoder. The decoder cannot directly access original pixel details, creating a bottleneck that limits the preservation of fine details, especially during tasks like image editing.
Internal+VPE Integration
This approach integrates VPE into internal architectures. It allows the diffusion model to simultaneously attend to text, reference images, and SigLIP 2 features through shared attention mechanisms at every decoding step. This direct cross-referencing of what to change (text) with original details (image) enhances editing preservation.

Terminology

Summary

The gist: Visual Prompt Engineering (VPE) improves image generation quality and editing preservation by inserting SigLIP 2 visual tokens as an intermediate semantic plan, which addresses the modeling difficulty between text and images.

How it works

  1. VPE introduces an intermediate semantic representation by first generating visual prompts Tvp (SigLIP 2 [8] tokens extracted from the target image) before generating the image tokens: c → Tvp z semantic plan → Timg z detail rendering (1)

  2. During training, visual prompts are obtained by encoding the ground-truth image I with a frozen visual semantic encoder, specifically SigLIP 2-Giant-Patch16-384 [8, 6] in this paper

  3. At inference time, the model autoregressively generates Tvp from c, then conditions image token generation on both c and the self-generated Tvp

Key Challenges and Solutions

** The external architecture introduces an information bottleneck that is particularly harmful for tasks requiring detail preservation such as image editing while abstracting away pixel-level details, making them an effective intermediate representation for semantic planning **

** The train–inference discrepancy can lead to severe quality degradation when the model must condition on its own predicted visual prompts, which are inevitably imperfect **

** In continuous-token models, the cross-entropy loss for predicting SigLIP 2 tokens and the flow-matching loss for image tokens operate at different scales, making joint optimization unstable **

VPE Training Strategies

  1. A progressive training schedule is introduced to gradually increase the model’s reliance on visual prompts to prevent degeneration into a simple copying task

  2. For discrete token models, a step-dependent masking probability pmask(s) is defined where visual prompt tokens are replaced with padding tokens with probability pmask, and no loss is computed on masked positions

  3. For continuous token models, the cross-entropy loss coefficient λCE(s) is modulated to prevent the cross-entropy gradient from overwhelming the diffusion objective in early training

Architectural Paradigms and Results

** Internal architectures avoid this bottleneck by enabling cross-modal interaction within a single model, but they have not incorporated visual semantic representations, leaving the difficult text-to-pixel modeling gap unaddressed **

** Internal+VPE enables the DiT model to directly attend to text, reference images, and SigLIP 2 features simultaneously through shared attention (Eq. 4), allowing cross-referencing of t (what to change) with x (original details) at every decoding step **

** External+VPE introduces a separate autoregressive model that generates SigLIP 2 tokens, which are then routed to a separate DiT decoder via cross-attention conditioning **

Validation and Impact

  1. VPE is validated across class-conditional generation, text-to-image generation, and image editing, showing consistent improvements across these tasks

  2. Results show that VPE can accelerate convergence, raise quality ceilings, and through internal integration (Internal+VPE), achieve substantially better editing preservation (PSNR: 26.76 vs. 19.92) than external alternatives of the same parameter scale

  3. Internal+VPE dramatically outperforms External+VPE in detail preservation during editing, with a 2.5× better Structure Distance (24.60 vs. 61.66) and +6.84 dB PSNR (26.76 vs. 19.92)

  4. VPE alleviates the difficulty of directly modeling text-to-image within a single model, accelerating the convergence of internal architectures while retaining their editing advantage

This work improves the quality and efficiency of image generation models, which carries both positive and negative societal implications. The paper encourages the community to develop and deploy appropriate safeguards, including watermarking and content provenance tools, alongside advances in generation quality. >

--- Page 1 ---

The gist: Visual Prompt Engineering (VPE) improves image generation quality and editing preservation by inserting SigLIP 2 visual tokens as an intermediate semantic plan, which addresses the modeling difficulty between text and images.

  1. VPE is validated across class-conditional generation, text-to-image generation, and image editing, showing consistent improvements across these tasks

  2. Results show that VPE can accelerate convergence, raise quality ceilings, and through internal integration (Internal+VPE), achieve substantially better editing preservation (PSNR: 26.76 vs. 19.92) than external alternatives of the same parameter scale

  3. Internal+VPE dramatically outperforms External+VPE in detail preservation during editing, with a 2.5× better Structure Distance (24.60 vs. 61.66) and +6.84 dB PSNR (26.76 vs. 19.92) <ref:

Improvements for AI systems

  1. The improved system can achieve substantially better editing preservation, specifically demonstrating PSNR: 26.76 vs. 19.92 compared to external alternatives of the same parameter scale, by leveraging internal architectures that allow cross-modal interaction within a single model.

  2. The system can accelerate convergence across both discrete and continuous token types by utilizing a progressive training schedule where it gradually increases the model’s reliance on visual prompts, addressing the train–inference discrepancy and loss-scale imbalance.

  3. The improved system can achieve superior text rendering quality, as shown by Show-o2+VPE achieving OCR accuracy improvements of 6.2× over Show-o2∗ on TextScenesHQ, confirming that visual prompts help the model “plan” to produce more accurate character structures during the subsequent generation phase.

  4. The system can generate higher quality images under comparable training budgets by showing that VPE substantially raises the quality ceiling, with LlamaGen+VPE achieving an FID of 8.69 vs. 24.56 at epoch 100 on ImageNet-1K without classifier-free guidance (CFG=1.0).

  5. The system can effectively utilize smaller models to outperform much larger ones, as shown by External+VPE surpassing all compared 7B+ models on overall score with only 4.16B parameters, illustrating that VPE reduces the modeling difficulty and improves generation quality.

Sources

Related papers