MIXAR: Scaling Autoregressive Pixel-based Language Models to Multiple Languages and Scripts

arXiv:2604.11575 · cs.CL · Submitted 2026-04-13 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: Today's paper: "MIXAR: Scaling Autoregressive Pixel-based Language Models to Multiple Languages and Scripts".

Jane: The gist The first generative and multilingual pixel-based LLM, MIXAR, is introduced to address tokenization challenges by utilizing rendered text as input across eight languages and multiple scripts,

Tom: First, who's behind it and why it matters.

Title and authors: Jane: So, moving onto the specifics of this paper, "MIXAR: Scaling Autoregressive Pixel-based Language Models to Multiple Languages and Scripts," who are we looking at here? Tom We’re seeing authors like Chen Hu, Yintao Tai, Antonio Vergari, Frank Keller, and Alessandro Suglia from the School of Informatics at the University of Edinburgh Jane That team has been working on this area for a while, so they bring a lot of experience to this visual language modeling space. Lu It’s notable that they are tackling the challenge head-on by proposing an autoregressive architecture instead of just a discriminative one Tom They’re focused on making sure the model can actually produce text, which is different from just classifying input.

Meng: I'm wondering about their background in large-scale training; how challenging was it to get this 0 point 5B parameter version trained on such a diverse multilingual dataset <ref:2604.11575#pg1>? Jane Well, they describe using a diverse multilingual dataset for pretraining, including the mC4 dataset Tom That corpus gives them domain diversity through web crawling across eight languages like German, English, Spanish, Italian, French, Korean and Chinese and Japanese Lu It’s a very broad set of inputs to start with.

Tom: And they detail their training process too. They mentioned an interleaved sampling strategy to maintain a balanced sample distribution across those languages during the training run Jane That’s smart; you don't want one language dominating the learning process while others get ignored Meng It shows they were thinking about how to manage that diversity from the start.

Lu: And they describe their training regime in two stages, which is where a lot of the technical depth comes in Tom Stage one is trained via maximum likelihood estimation using teacher forcing to minimize the negative log-likelihood of ground-truth pixel patches Jane Then Stage two introduces an adversarial pretraining stage where MIXAR stage one acts as a generator G within a generative adversarial network framework to stabilize training Meng That GAN part sounds like they’re trying to get the model into a more stable state, which is always tricky when you're dealing with generative models.

Tom: And they mention their architectural choices too. They increased the operational patch size from eight by eight pixels to thirty-two by thirty-two pixels Jane That change was specifically done to support logographic scripts like Chinese, Japanese, and Korean Lu So the architecture had to grow alongside the complexity of the data they were feeding it.

Jane: It shows they’re not just tweaking a model; they are fundamentally changing how it processes language by adapting its visual resolution for different writing systems Meng That’s a significant architectural decision when you're thinking about multilingual generalization.

Tom: And before we move on, I want to make sure everyone has had a chance to react to what we’ve covered so far. The team is really focused on how they built this thing from the ground up Jane Next up, we're going into the actual summary of what MIXAR actually does Lu

The paper's summary: Jane: Now that we’ve talked about who they are and how they trained it, let’s get into what this whole paper is actually summarizing about the MIXAR model. Tom So at its core, the summary is that MIXAR introduces a generative pixel-based language model that can handle multiple languages and scripts simultaneously Jane It’s designed to overcome the tokenization problems by using rendered text as input across eight different languages and multiple scripts Lu They are demonstrating substantial performance improvement over previous models in both discriminative and generative tasks Tom

Meng: So what does "substantial performance improvement" mean in concrete terms? Are we talking about beating GPT-two on a specific task or something more general? Jane They show that it’s better than prior pixel-based models, and importantly, it’s competitive with tokenizer-based models like BERT in several settings Tom

Lu: The paper points out that the main innovation here is moving from just having a masked auto-encoder to achieving something generative, as they propose the first generative and multilingual pixel-based LLM Jane That’s paving the way towards systems that dispense with tokenizers entirely Meng They are aiming for something more realistic.

Tom: So, they are showing that by using pixels, you get a direct representation of language from rendered text, which overcomes weaknesses like being prone to vocabulary attacks Jane That's a major win in terms of robustness.

Jane: And they show the model is also robust to languages it was never seen during training Meng That means the transfer ability across different languages is much stronger than what you see in older models like PIXEL or PIXAR, which were trained only on English Lu

Tom: That cross-lingual transferability is a key result they are highlighting here, showing that the model learns things that apply broadly Jane It’s not just about mastering one language.

Meng: So for us in production, what does this mean for deployment? Does it mean we can deploy it anywhere without worrying about needing language-specific fine-tuning? Jane It suggests a lot of flexibility, but they also noted that performance isn't always consistent for scripts that are quite different from the ones they trained on during pretraining Tom

Lu: That’s a necessary caveat; you can’t just deploy it blindly and expect perfection everywhere yet Jane They also pointed out that generative performance still lags behind some of the best token-based baselines in certain scenarios Meng

Tom: So, they’re showing a strong capability here, but it's not claiming to solve every single NLP problem with just this one model structure.

Jane: Overall, they’ve laid the foundation for a more inclusive way to build language technologies using visual information Lu It’s an important step in that direction.

The paper's improvements: Tom: Okay, let’s look at the specific improvements they list and unpack what those mean for us as an engineer. Jane The paper points out several key architectural changes that make MIXAR better than its predecessors Tom They increased the operational patch size from eight by eight pixels to thirty-two by thirty-two pixels Lu This was specifically to support logographic scripts like Chinese, Japanese, and Korean Meng So they had to physically change how the model looked up text.

Jane: That means they’re optimizing the visual input format for specific script types rather than just using one fixed resolution for everything Tom And they incorporate rotary positional embeddings and SwiGLU activation functions, following architectural refinements from models like LLaMA-two Lu These are standard but important details that show they’re building on proven foundations.

Meng: I'm curious about the activation functions; what do those actually do for the model’s training stability? Jane SwiGLU is used because it helps with training stability, and they also use pre-normalization using RMSNorm, which is another technique that helps keep things steady during optimization Tom

Lu: The adversarial pretraining stage is a crucial improvement because it stabilizes the overall learning process Jane It’s essentially acting as a safety net during the training process to make sure the model doesn't just collapse into something less useful Meng That regularization is essential for complex generative models.

Tom: And they explicitly mention how this architecture lets them overcome weaknesses like being prone to vocabulary attacks because language is encoded directly from rendered text Jane That’s a big win for security if you're worried about adversarial inputs Lu They are leveraging the visual nature of the input to their advantage.

Jane: And they show that scaling it up to 0 point 5B parameters significantly improves its capabilities in generative tasks like LAMBADA, where it actually outperforms PIXAR and GPT-two on average across those multilingual generative tasks Meng That’s a tangible result when you look at the numbers Tom

Lu: And they also show robustness against visual attacks when facing orthographic attacks, where the model is shown to be more robust than baseline models Jane So, it seems the architecture and scaling together really work to build a tougher system.

Meng: I’m still thinking about deployment constraints. If we scale it further, what breaks at scale? Tom The paper itself doesn't get into the specifics of the infrastructure required for those larger models Jane But we know that moving to bigger parameter counts always increases computational needs significantly Lu

Tom: So they’re proving that this pixel-based approach can be scaled up and handle complexity better than previous token-based methods did Jane That’s the core message here.

Conclusion: Jane: Alright, we’ve covered a lot about "MIXAR: Scaling Autoregressive Pixel-based Language Models to Multiple Languages and Scripts." To wrap up, what’s the final word on its implications for the field? Tom The main implication is that multilingual pixel-based language models can be extended beyond English to a diverse set of languages and scripts by increasing patch resolution, scaling the architecture up to 0 point 5B parameters, and pretraining on eight languages spanning Latin and CJK scripts Lu It consistently improves over prior pixel-based approaches Jane And it achieves competitive results against tokenizer-based models in several settings Meng <ref:2604.11575#pg1>

Tom: It shows that this is a promising direction for building more inclusive and resilient language technologies Jane It’s a step toward systems that dispense with tokenizers entirely if we want to move forward.

Lu: I think the ability to leverage visual information directly from rendered text is what makes this model uniquely positioned for cross-lingual tasks Meng It shows the unique advantages and limitations of a pure visual architecture in cross-lingual tasks Jane

Jane: So, even with the noted limitations regarding script variation, this work provides a solid blueprint for how to build more inclusive language systems Tom We've seen how MIXAR can be scaled up effectively to handle complexity across different languages and scripts Lu

Meng: For deployment, we still need to see more evidence that it holds up when it’s actually running on production traffic that isn't perfectly clean Tom That’s the next hurdle.

Jane: It sounds like the path forward is about proving this approach works in more messy, real-world scenarios Lu But for now, "MIXAR: Scaling Autoregressive Pixel-based Language Models to Multiple Languages and Scripts" gives us a solid direction to consider.

University of Edinburgh

cs.CL

Submitted: 2026-04-13

Updated: 2026-10-15

Comments: 29 pages. Code and models: https://github.com/april-tools/mixar

Code: https://github.com/april-tools/mixar

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 85/100

The gist: The gist The first generative and multilingual pixel-based LLM, MIXAR, is introduced to address tokenization challenges by utilizing rendered text as input across eight languages and multiple

Key concepts

Pixel-based Language Models: These are language models that process text not as discrete tokens but as a grid of pixels derived from rendered characters. This method aims to bypass the limitations of traditional tokenization by using the visual structure of text directly, allowing a single model to potentially handle many languages and scripts without needing massive embedding size increases.
Multilingual Pretraining: MIXAR was trained on a diverse dataset covering eight languages and multiple writing systems, including Latin and CJK scripts. This broad training ensures the model can generalize its knowledge across different linguistic structures, making it better at handling unseen low-resource languages compared to models trained only on English.
GRPO-based Reinforcement Learning: Instead of unstable adversarial training, MIXAR uses a reinforcement learning stage based on GRPO. This stage rewards the model for generating text that runs smoothly without interruption. This technique made the free generation process much more stable and significantly improved accuracy on multilingual benchmarks.
Patch Size Scaling: The architecture was enhanced by increasing the operational patch size from 8x8 to 32x32 pixels. This change was necessary to properly support complex logographic scripts like Chinese, Japanese, and Korean characters within the model's visual processing layers.

Terminology

Generalization — 1× in this paper · explained in 13 episodes
Robustness — 1× in this paper · explained in 5 episodes
Tokenization — 1× in this paper · explained in 2 episodes

Summary

The gist The first generative and multilingual pixel-based LLM, MIXAR, is introduced to address tokenization challenges by utilizing rendered text as input across eight languages and multiple scripts, demonstrating substantial performance improvements over previous models in both discriminative and generative tasks

Motivation for Pixel-Based Models

Pixel-based language models are proposed as an alternative to traditional token-based approaches because a single pixel-based model should be able to handle multiple languages and scripts without the need to increase its embedding size This approach leverages visual information directly from rendered text, overcoming weaknesses of tokenizer-based language models such as being prone to vocabulary attacks While previous models like PIXEL and PIXAR were trained only on English, MIXAR is the first generative and multilingual pixel-based LLM designed to pave the way towards more realistic NLP systems that dispense with tokenizers

Architecture and Scaling of MIXAR

MIXAR is an autoregressive decoder-only generative LLM that comprises a stack of transformer layers, similar to PIXAR but with key enhancements Key architectural refinements include increasing the operational patch size from 8×8 to 32×32 pixels to support logographic scripts like Chinese, Japanese, and Korean The architecture is scaled to accommodate this larger patch size, with experiments conducted on both a smaller model with 116M parameters and a larger 477M parameter model featuring 24 Transformer layers MIXAR incorporates enhancements such as rotary positional embeddings, SwiGLU activation functions, and pre-normalization using RMSNorm, following architectural refinements introduced in LLaMA-2

Multilingual Pretraining and Training Regime

MIXAR is trained on a diverse multilingual dataset to ensure broad generalization across languages and scripts The pretraining datasets utilized include the mC4 dataset, which introduces domain diversity through web crawling, encompassing eight languages including German, English, Spanish, Italian, French, Korean, Chinese and Japanese To maintain a balanced sample distribution across these languages during training, an interleaved sampling strategy is employed The training regime involves two stages: Stage 1 is trained via maximum likelihood estimation (MLE) using teacher forcing to minimize the negative log-likelihood of ground-truth pixel patches Stage 2 introduces an adversarial pretraining stage where MIXAR stage1 acts as a generator G within a generative adversarial network (GAN) framework to stabilize training

Experimental Evaluation and Performance

MIXAR was evaluated across both discriminative and generative NLP tasks, demonstrating performance competitive with tokenizer-based models like BERT and superior to current state-of-the-art pixel-based models Evaluations include:

Discriminative tasks on the GLUE benchmark, where MIXAR stage2 (477M) achieved performance competitive with GPT-2 and outperformed MGPT-2 on seen and unseen languages on XNLI
Multilingual discriminative benchmarks like XNLI and SIB-200, where MIXAR stage2 (477M) showed a Seen AVG of 86.4, outperforming the text-based MGPT-2
Generative tasks such as bAbI and LAMBADA, where scaling MIXAR to 477M outperforms PIXAR and GPT-2 on average across multilingual generative tasks
Robustness against visual attacks, where MIXAR models are shown to be more robust than baseline models when facing orthographic attacks

Robustness and Limitations

MIXAR exhibits robustness to input perturbations such as orthographic attacks, with performance dropping significantly only at higher attack ratios However, the paper notes that generative performance still trails strong token-based baselines in several settings, and the gains from the second GAN-based training stage are not consistently realized Furthermore, performance remains uneven for scripts that differ substantially from those observed during training The authors plan to investigate retrieving dependencies to improve generation by replacing the final layer with a circuit or a diffusion model in future work

Conclusion

MIXAR demonstrates that multilingual pixel-based language models can be extended beyond English to a diverse set of languages and scripts through increasing patch resolution, scaling the architecture up to 477M parameters, and pretraining on eight languages spanning Latin and CJK scripts This approach consistently improves over prior pixel-based approaches, achieves competitive results against tokenizer-based models in several settings, and shows promising transfer to languages not seen during pretraining The findings suggest that multilingual pixel-based language models are a promising direction for building more inclusive and resilient language technologies

Acknowledgment

This research was funded in part by the UKRI AI Centre for Doctoral Training in Responsible and Trustworthy in-the-world Natural Language Processing (EP/Y030656/1) Antonio Vergari was supported by the “UNREAL: Unified Reasoning Layer for Trustworthy ML” project (EP/Y023838/1) selected by the ERC and funded by UKRI EPSRC This work used the Cirrus UK National Tier-2 HPC Service at EPCC (http://www.cirrus.ac.uk) funded by the University of Edinburgh, the Edinburgh and South East Scotland City Region Deal, and UKRI via EPSRC Additionally, this work was supported by the Edinburgh International Data Facility (EIDF) and the Data-Driven Innovation Programme at the University of Edinburgh

Reproducibility Statement

To ensure reproducibility of the findings in this paper, all components will be released including model architecture and hyperparameters, training data and multilingual corpus details, evaluation protocols for discriminative and generative tasks, perturbation methodology for orthographic attacks, source code with both training and inference implementations, pre-trained checkpoints for models to facilitate immediate testing, a rendering pipeline package for text-to-pixel conversion, and computational resource usage statistics The pretraining of MIXAR 116M was completed in ∼4 days using 16 NVIDIA V100 GPUs on an HPC computing cluster by extending the codebase originally released by Tai et al. (2024) Whereas, the pretraining of MIXAR 477M was completed using 8 nodes containing 32 V100 NVIDIA GPUs in ∼20 days Finetuning experiments are shorter and use the same computing infrastructure The pretraining dataset information is detailed in Table 13, showing the number of patches used for stage 1 training across various benchmarks The rendering process details font selections, font sizes, and normalization techniques to ensure consistency across scripts are documented in Table 5 The performance evaluation protocols specify the exact versions of tasks used on Huggingface and provide explicit links to the benchmarks in the codebase The process for generating orthographic attacks is clearly defined to allow for independent robustness testing following previous work from Tai et al. (2024) The source code will include the full implementation of MIXAR using PyTorch and Transformers, including both training and inference code Pre-trained checkpoints for the models will be provided to facilitate immediate testing and downstream fine-tuning The rendering pipeline scripts used to transform raw text into the pixel-based format required by the model will be released as a standalone package The pretraining of MIXAR 477M was completed using 8 nodes containing 32 V100 NVIDIA GPUs in ∼20 days Finetuning on 10 XNLI tasks took 48.

Improvements for AI systems

The MIXAR model can handle multiple languages and scripts by encoding language as pixels, addressing a significant hurdle for multilingual generalization in pixel space. This enables systems to be robust to visual attacks as well, which is crucial for deployment in real-world scenarios where input perturbations are common.
Scaling the architecture up to 477M parameters significantly improves generative capabilities, as scaling improves not only generative performance, especially on multilingual LAMBADA, but also robustness to orthographic perturbations. This allows the improved AI system to generate high-quality text completions across diverse languages and resist attacks that replace individual letters with visually similar ones.
The model exhibits enhanced cross-lingual transferability, as the English-only PIXAR stage2 achieves a strong score of 78.8 on the English task; however, its performance degrades significantly when processing non-English languages. This means the improved AI can leverage knowledge from one language to perform well on others, even those not seen during pretraining.
The system can be made more robust to font variations by investigating whether MIXAR is robust to changes in the font used, which addresses a vulnerability where performance drops to almost zero when the attack ratio is 0.5 when the font changes. This would allow for more reliable output regardless of minor rendering inconsistencies in input text.
The improved AI system can perform better on tasks requiring low-level linguistic understanding, as MIXAR demonstrates the unique advantages and limitations of a pure visual architecture in cross-lingual tasks and achieves highly competitive performance on zero-shot cross-lingual tasks.

Abstract

Pixel-based language models are gaining momentum as alternatives to traditional token-based approaches, promising to circumvent tokenization challenges; however, existing models remain English-centric and are hard to train as generators. This paper introduces MIXAR, the first generative multilingual pixel-based language model, pretrained on eight high-resource languages spanning Latin and CJK scripts and scaled up to 0.5B parameters. Although pretrained on high-resource languages only, MIXAR generalizes to unseen low-resource languages better than a tokenizer-based model pretrained on the same corpus, with its largest gains on the scripts most distant from the pretraining mix, taking a step toward more equitable language technology. To make free generation work in pixel space, MIXAR replaces the notoriously unstable adversarial training stage of prior pixel-based models with a GRPO-based reinforcement-learning stage that rewards the model on its own free-running generations: it proves more stable and multiplies free-generation accuracy by up to 2.3x on multilingual benchmarks. Finally, MIXAR performs on par with comparable tokenizer-based models on discriminative benchmarks, improves consistently with scale, and withstands orthographic attacks more gracefully than tokenizer-based models. Code and models are available at https://github.com/april-tools/mixar.