Fine-tuning Multi-modal LLMs with ART: Art-based Reinforcement Training

arXiv:2606.11854 · cs.LG, cs.AI, cs.CL · Submitted 2026-06-10 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "Fine-tuning Multi-modal LLMs with ART".

Jane: Here's the gist of the paper. Fine-tuning Multi-modal LLMs with ART: h ART-BASED REINFORCEMENT TRAINING is a method that injects information into a frozen Multimodal Large Language Model by optimizing only its raw visual input, thus enabling the softtoken approach on pre-compiled computational graphs The method relies on backpropagation of gradients back into a plain pixel array and thus supports any fine-tuning objective Modern LLMs have evolved from text-only generators to multimodal agents capable of processing combined text, image, and video inputs out of the box Despite this inherently multimodal capability, a vast majority of downstream tasks remain formulated as purely text-based instructions Developing efficient fine-tuning methods remains an important research direction.

Tom: The ART Framework ART repurposes the visual input channel of modern multi-modal LLMs as a non-invasive interface for task adaptation The method adapts the model by optimizing a single task-specific ART input image routed through the standard vision pathway The model remains completely frozen, and fine-tuned prompts are treated by the serving infrastructure as plain multi-modal requests.

Jane: Pixel-Space Parameterization The theoretical intuition behind this approach is that the Vision Transformer (ViT) and cross-modal projection layer act as a frozen, pre-aligned continuous mapping from raw pixel coordinates to the model’s embedding space The learnable image in logit space is parameterized to keep pixel values strictly in the valid range while allowing unconstrained optimization At any point during training, the 8-bit image is recovered by quantizing the sigmoid.

Tom: Reward-Driven Optimization The optimization target Xraw (the raw logit-space pixel parameters) is agnostic to the training algorithm, meaning any procedure that backpropagates a scalar loss can be substituted without changing the artifact or the serving path The optimization pipeline operates in a custom two-pass loop per step, decoupling the high-throughput rollout generation from the backpropagation engine.

Paper summary: Tom: So we're looking at this paper called "Fine-tuning Multi-modal LLMs with ART: Art-based Reinforcement Training." Basically, they're trying to figure out how to fine-tune these big multimodal language models without having to mess with the entire model structure.

Jane: Right, so what's their main idea here? What are they actually proposing?

Tom: They propose this method called ART, which is Art-based Reinforcement Training. The core idea is that instead of tweaking the whole model, you only optimize the raw visual input part of a frozen multimodal LLM.

Lu: That means the rest of the model stays completely frozen while they are fine-tuning just that image input.

Jane: That sounds like it could be really helpful because most people use these models for text tasks, right? So, what's the actual claim they are making about this ART approach? What do they say it achieves?

Tom: They claim that by optimizing only the raw visual input, you enable a soft token approach on pre-compiled computational graphs. This lets them fine-tune models even when the standard methods like LoRA or Soft Prompting aren't fully supported in high-throughput engines like vLLM <ref:2606.11854#pg1>.

Meng: Hold on, if it's optimizing just the raw visual input, what does that actually look like in practice? How do you use this method to teach the model a new task? I mean, what's the actual process for getting it working?

Jane: Well, they describe it as repurposing the visual input channel of these multimodal LLMs as a non-invasive interface for task adaptation. They adapt the model by optimizing a single task-specific ART input image routed through the standard vision pathway <ref:2606.11854#pg2>.

Lu: The theoretical backing for this is that the Vision Transformer and that cross-modal projection layer act like a frozen, pre-aligned continuous mapping from raw pixel coordinates to the model’s embedding space. They then parameterize the learnable image in logit space so it stays within valid pixel ranges while allowing optimization <ref:2606.11854#pg2>.

Tom: And they mention something about reward-driven optimization, which is a key part of how it works. The optimization target, they call Xraw, is agnostic to the training algorithm; any procedure that backpropagates a scalar loss can be substituted without changing the serving path <ref:2606.11854#pg2>.

Paper summary: Jane: That sounds like flexibility, which is good because you don't have to tie it down to just one specific loss function for every new task. What kind of tasks are they testing this on?

Lu: They tested it on three benchmarks: GSM8K, GPQA, and ToolMind. The results show that the optimized image reaches fifty-eight point five three percent on GSM8K plus an eighteen point eight eight percent over the baseline <ref:2606.11854#pg2>.

Tom: And for ToolMind, it gets seventy-three point eight zero percent over the baseline with a ten point seven zero percent improvement over random image <ref:2606.11854#pg2>. They also compared it to LoRA on both the 0 point 8B and 2B models, and ART actually edges LoRA on the smaller model, getting seventy-three point eight zero percent versus sixty-nine point five zero percent <ref:2606.11854#pg2>.

Meng: Wait, so it’s faster than LoRA in terms of training time? That's interesting because for us at scale, training time is a big factor when we have to run these things on production hardware <ref:2606.11854#pg3>.

Jane: They actually state that ART trains roughly twice as fast as LoRA on GSM8K and more than three times faster on ToolMind <ref:2606.11854#pg3>. That efficiency is something they are really pushing because it makes sense for locally-served small LLMs where you can't afford massive fine-tuning overhead.

Lu: The paper notes that the defining property of ART is that the learned artifact is a native input to a pre-optimized MLLM, and those fine-tuning artifacts can be considered a form of computational art. They even point out how the growth in PNG file size indicates task information storage in images, making ART a kind of steganography for AI <ref:2606.11854#pg2>.

Tom: So it’s not just adding weights; it’s storing task knowledge in an image format that the model already understands through its visual pipeline. That's a pretty neat way to think about information storage in this context.

Jane: It really shifts the focus from traditional token manipulation to something more visual, which is interesting because we see how much influence the vision channel has on text generation <ref:2606.11854#pg3>. This paper seems to be showing that you can use that visual channel for enhancement rather than just fighting those security vulnerabilities they found with adversarial images <ref:2606.11854#pg3>.

Meng: I worry about deploying this on real traffic, though. They're training on specific seed images—a math book for GSM8K, a brain for GPQA, and tools for ToolMind <ref:2606.11854#pg2>. How reliable is that generalization when we run it on actual user data?

Paper summary: Lu: The paper focuses exclusively on the Qwen3 point 5 architecture family, specifically the 0 point 8B and 2B models in their experiments <ref:2606.11854#pg1>. They acknowledge that generalizing this method to additional architectures is an important direction for future work <ref:2606.11854#pg3>.

Tom: So, the current limitation is that it's not yet proven to work reliably across different model sizes or architectures. But they are also exploring whether the training objective itself matters, contrasting their policy-gradient setup against supervised fine-tuning <ref:2606.11854#pg3>.

Jane: That’s a fair limitation—it’s still mostly confined to the Qwen family right now, which is where they did all their hard comparisons. But what do we get from this for someone who just uses these models every day? What does this mean for the everyday user?

Lu: It suggests a path toward making fine-tuning more efficient and less dependent on complex, heavy adapter layers that require custom graph captures per active adapter count <ref:2606.11854#pg2>. It’s about injecting information in a way that respects the existing, pre-optimized structure of the model.

Tom: So, to summarize this paper "Fine-tuning Multi-modal LLMs with ART: Art-based Reinforcement Training," they propose using Art-based Reinforcement Training to fine-tune frozen multimodal LLMs by optimizing only their raw visual input <ref:2606.11854#pg1>. They show this method achieves performance gains on GSM8K, GPQA, and ToolMind compared to baseline methods like LoRA and Soft Prompting <ref:2606.11854#pg2>, while being significantly faster in training time <ref:2606.11854#pg3>.

Jane: And the conclusion they draw is that this method turns the visual channel into a way to enhance model capabilities, rather than just using it to find vulnerabilities <ref:2606.11854#pg3>. It’s about storing task information in images, which they call steganography for AI <ref:2606.11854#pg2>.

Lu: And they point out that the method's limitation is currently that it's confined to the Qwen3 point 5 architecture family, and they plan on exploring whether their policy-gradient setup actually performs better than supervised fine-tuning <ref:2606.11854#pg3>. That’s a clear boundary for what they’ve shown so far.

Tom: So, the main implication right now is that if you have a small multimodal LLM and you need task adaptation without having to modify the core computation graph, ART offers a way to inject information via raw visual input <ref:2606.11854#pg1>.

Paper summary: Jane: It means for users who are building local setups or running models on smaller hardware, there might be a more efficient way to adapt them than those methods that require constant custom engineering of embedding prepending <ref:2606.11854#pg3>.

Lu: The paper is essentially exploring how continuous soft prompting can be extended into this pixel space for adaptation <ref:2606.11854#pg3>. It’s about finding a way to leverage the visual channel more effectively for fine-tuning without needing full adapter weight loading <ref:2606.11854#pg3>.

Meng: From an engineering standpoint, the speed advantage is what really grabs my attention—training twice as fast on GSM8K and three times faster on ToolMind <ref:2606.11854#pg3>. That efficiency could make a huge difference when we're deploying these things at scale, even if it’s only for locally served models.

Tom: Exactly, the number is what matters here—that speed and performance comparison against LoRA shows that ART has real traction in this space <ref:2606.11854#pg2>. The fact that they are decoupling the rollout generation from backpropagation engine also means this setup is viable for high-throughput environments <ref:2606.11854#pg3>.

Jane: So, while they are still working on generalizing it beyond Qwen architectures, the immediate value is in showing a novel way to use visual input as a trainable parameter for fine-tuning <ref:2606.11854#pg2>. It’s a new lens through which we can look at adapting these multimodal agents.

Lu: And they’ve established that while Soft Prompting requires custom engineering to prepend continuous embeddings, ART does it by optimizing the raw pixel parameters in a way that respects the existing structure <ref:2606.11854#pg3>. That’s a different kind of injection mechanism.

Tom: So, if you're curious about how this works mechanistically—how they map pixel values to logit space and back—that’s where the real engineering magic is happening <ref:2606.11854#pg2>.

Jane: That sounds like a deep dive, but for now, the main thing to remember is that they’ve found a method that lets you fine-tune multimodal models by optimizing just one image input using this ART framework <ref:2606.11854#pg1>.

Meng: And for deployment, it sounds promising because it avoids loading extra adapter weights on top of the main model, which is a real win when you’re trying to keep memory usage low <ref:2606.11854#pg3>.

Lu: The future work they mentioned is important—they want to explore ablating the training objective itself, contrasting their policy-gradient setup against supervised fine-tuning <ref:2606.11854#pg3>. That will help them understand the specific mechanism behind why their setup works better than

Conclusion: Tom: So, we're wrapping up this look at "Fine-tuning Multi-modal LLMs with ART: Art-based Reinforcement Training."

Jane: It’s about taking a frozen multimodal LLM and fine-tuning it by optimizing just the raw visual input image using this ART method.

Tom: The authors are focusing on how you can inject task information into that model without having to touch the core architecture or load any massive adapter weights on top.

Lu: They’re proposing that because the vision pathway is already pre-aligned, you can parameterize the learnable input image in a way that keeps pixel values valid while still allowing optimization.

Meng: And they show it works better than LoRA on some of these benchmarks, specifically showing faster training times on models like ToolMind.

Jane: The authors are basically saying that this approach lets you store task-specific knowledge right inside the input image, almost like steganography for AI.

Tom: It’s about making fine-tuning more efficient for smaller multimodal models because you aren't loading a whole new set of weights every time.

Jane: It means for people who are building local setups or running models on smaller hardware, there might be a more practical way to adapt them without needing heavy custom engineering.

Lu: The limitation they point out is that right now, they’re only testing this on the Qwen3 point five family of models <ref:2606.11854#pg1>.

Meng: That means we need to see if this actually holds up when we move to different model architectures, which is definitely the next step in their work.

Tom: So, while it’s still focused on Qwen for now, they’re proving that optimizing just the visual input can be a powerful way to adapt these models.

Jane: And that opens up a lot of questions about how much information we can actually store visually in an image before it starts losing meaning for the model.

University of Stavanger · NORCE Research

cs.LG, cs.AI, cs.CL

Submitted: 2026-06-10

Updated: 2026-10-29

Code: https://github.com/jinymusim/ART

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 80/100

The gist: Fine-tuning Multi-modal LLMs with ART: h ART-BASED REINFORCEMENT TRAINING is a method that injects information into a frozen Multimodal Large Language Model by optimizing only its raw visual input,

Terminology

Summary

Fine-tuning Multi-modal LLMs with ART: h ART-BASED REINFORCEMENT TRAINING is a method that injects information into a frozen Multimodal Large Language Model by optimizing only its raw visual input, thus enabling the softtoken approach on pre-compiled computational graphs The method relies on backpropagation of gradients back into a plain pixel array and thus supports any fine-tuning objective

Introduction to the Problem

Modern LLMs have evolved from text-only generators to multimodal agents capable of processing combined text, image, and video inputs out of the box Despite this inherently multimodal capability, a vast majority of downstream tasks remain formulated as purely text-based instructions Developing efficient fine-tuning methods remains an important research direction.

The ART Framework

ART repurposes the visual input channel of modern multi-modal LLMs as a non-invasive interface for task adaptation The method adapts the model by optimizing a single task-specific ART input image routed through the standard vision pathway The model remains completely frozen, and fine-tuned prompts are treated by the serving infrastructure as plain multi-modal requests.

Pixel-Space Parameterization

The theoretical intuition behind this approach is that the Vision Transformer (ViT) and cross-modal projection layer act as a frozen, pre-aligned continuous mapping from raw pixel coordinates to the model’s embedding space The learnable image in logit space is parameterized to keep pixel values strictly in the valid range while allowing unconstrained optimization At any point during training, the 8-bit image is recovered by quantizing the sigmoid.

Reward-Driven Optimization

The optimization target Xraw (the raw logit-space pixel parameters) is agnostic to the training algorithm, meaning any procedure that backpropagates a scalar loss can be substituted without changing the artifact or the serving path The optimization pipeline operates in a custom two-pass loop per step, decoupling the high-throughput rollout generation from the backpropagation engine.

Performance Comparison

Table 1 summarizes the complete performance comparison across the three benchmarks for all model setups The Optimized Image (ART) reaches 58.53% on GSM8K (+18.88% over Baseline, +3.94% over random image) and 73.80% on ToolMind (+37.15% over Baseline, +10.70% over random image) ART leads LoRA on 0.8B (73.80% vs. 69.50%) and edges LoRA on 2B (69.05% vs. 67.15%) within overlapping confidence intervals.

Information Storage in Art

The defining property of ART is that the learned artifact is a native input to pre-optimized MLLM The resulting fine-tuning artifacts can be considered a form of computational art The growth in PNG-file-size indicates task-information storage in images, making ART a form of steganography for AI.

Efficiency and Conclusion

ART trains roughly twice as fast as LoRA on GSM8K and more than three times as fast on ToolMind ART performs faster than LoRA because the model stays frozen and no adapter weights need to be loaded This performance efficiency makes ART specifically attractive for locally-served small MLLMs.

Limitations and Future Work

Our experiments are conducted exclusively on the Qwen3.5 architecture family (0.8B and 2B) Generalizing ART to additional architectures is an important direction for future work. The optimization target is just an input, so we also intend to ablate the training objective itself, contrasting our policy-gradient (DAPO) setup against supervised fine-tuning.

Appendix Details

The image size of 256 × 256 maps to exactly 64 continuous visual prefix tokens Our random-string control pre-pends a matched budget of 64 text tokens, and a soft-prompting baseline of 64 learnable embeddings is left to future work. The Qwen processor patchifies the image into a 16 × 16 = 256 grid of patches (using a 16px patch size). The visual output sequence is thus flattened into the following Nvisual = H/16 × W/16 × 4 = 64 tokens. The image is serialized as a standard 8-bit PNG (final.png), which retains only 256 discrete levels per channel.

Benchmark Details

GSM8K consists of high-quality grade-school math word problems, requiring multi-step arithmetic conceptualization GPQA is an exceptionally difficult benchmark comprised of multiple-choice science questions authored by domain experts ToolMind evaluates an LLM’s capacity to construct structured XML-form API function calls based on environmental parameters. The scorer extracts the generated function name and arguments, and performs a strict match against the ground-truth parameters.

Experimental Setups

We run a dense grid of baselines across the Qwen3.5-0.8B and Qwen3.5-2B models on an NVIDIA A100 GPU The Optimized Image (ART) is trained via Algorithm 1 with the DAPO loss (LR = 0.1, AdamW, constant-withwarmup schedule, warmup for 5 steps). Both LoRA and ART were fine-tuned for 100 steps with a group size of 8 and an effective batch size of 32.

Results Summary

Table 2 shows that every single optimized image exhibits a substantial increase in raw compressed size The physics/math task-image for Qwen3.5-0.8B grows from a lean 8.5 KB to a heavy 98.0 KB (+1047% increase).

Acknowledgments and Declarations

This work is part of the Center for Research-based Innovation DigiWells, which stands for Digital Well Center for Value Creation, Competitiveness and Minimum Environmental Footprint During the preparation of this work, the authors used ChatGPT to draft some of the original paragraphs from the authors’ notes. The center is a cooperation of NORCE Norwegian Research Centre, the University of Stavanger, the Norwegian University of Science and Technology (NTNU), and the University of Bergen. The center is funded by Aker BP, ConocoPhillips, Equinor, Harbour Energy, Petrobras, TotalEnergies, Vår Energi, and the Research Council of Norway.

Improvements for AI systems

  1. Adaptive task specialization of multimodal LLMs by optimizing only raw visual input allows for fine-tuning without modifying model weights or runtime engines, making it significantly more efficient than LoRA during training and inference. This enables locally served small MLLMs to achieve performance gains competitive with or exceeding weight-space tuning on mathematics and structured tool use benchmarks.

  2. Generation of computational art that simultaneously encodes model fine-tuning, using growth in lossless PNG file size as a proxy for stored information, providing a tangible measure of task signal storage within the input image artifact. This suggests ART functions as a steganography for AI where optimized images serve as portable and compressible representations of learned behaviors.

  3. Improved reasoning capabilities on Qwen3.5-0.8B models on mathematics and tool use tasks by achieving accuracy competitive with LoRA across these benchmarks, surpassing text baselines when prepending an optimized image is used instead of random or text tokens. This demonstrates that the activation of the ViT decoder that improves attention routing is a key mechanism for task-specific performance boosts.

  4. Enhanced efficiency in high-throughput serving environments like vLLM by bypassing the architectural limitations of traditional PEFT methods, as ART requires no custom weight managers, no specialized kernels, and no architectural workarounds. This eliminates the overhead associated with dynamically loading weights or rebuilding CUDA graphs for different adapters during concurrent user sessions.

  5. Creation of task-specific visual prompts that can be leveraged for Generative Art and potentially for novel security research, as the visual channel exerts an outsized influence on text generation, making optimized images a potential vector for both enhancement and adversarial reprogramming.

Sources

Related papers