Giraffe: A Mapping Architecture from Hidden Text Representations to Visual Embeddings for Efficient Graphic Design
cs.AI, cs.LG
Submitted: 2026-08-25
Updated: 2026-08-25
License: http://creativecommons.org/licenses/by/4.0/
The gist: Multimodal large language models (MLLMs) have made significant progress in understanding and interpreting mul- timedia content.
Terminology
Abstract
Multimodal large language models (MLLMs) have made significant progress in understanding and interpreting mul- timedia content. However, their ability to generate me- dia remains limited. Recent approaches have attempted to bridge this gap by translating the hidden representations of token sequences into the embedding space of visual models or directly into raw image data. However, these methods often represent each image using multiple specialised to- kens which significantly increases the input length. This be- comes a major limitation for tasks such as graphic design generation where the output typically involves a seamless blend of thousands of tokens across text, multiple images, and layout information. To address this challenge, a novel architecture is proposed that maps hidden token represen- tations to the embedding space of visual models, such as CLIP ViT-L/14, using a single [IMG] token per image. The architecture employs two shallow MLP blocks, each with a separate compression module followed by a shared expan- sion module, trained with six distinct loss functions. One block aids the other during training and is omitted during inference, resulting in a lightweight solution. Strong perfor- mance is demonstrated in both image-to-design and text-to- design generation tasks.
Sources
- TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
- DreamLLM: Synergistic Multimodal Comprehension and Creation
- GPT-4o System Card
- Auto-Encoding Variational Bayes
- Representation Learning with Contrastive Predictive Coding
- DreamFusion: Text-to-3D using 2D Diffusion
- Emu: Generative Pretraining in Multimodality
- Chameleon: Mixed-Modal Early-Fusion Foundation Models
- Gemma 3 Technical Report
- Towards Perceiving Small Visual Details in Zero-shot Visual Question Answering with Multimodal LLMs
- Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Model
Related papers
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Agents' Last Exam
- The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
- MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
- Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems