Lizard: An Efficient Linearization Framework for Large Language Models

arXiv:2507.09025 · cs.CL, cs.LG · Submitted 2025-07-11 · Read on arXiv

cs.CL, cs.LG

Submitted: 2025-07-11

Updated: 2026-04-18

Comments: ACL 2026 (Main)

Journal ref: Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 34935-34948 (2026)

DOI: 10.18653/v1/2026.acl-long.1613

Code: https://github.com/EleutherAI/lm-evaluation-harness

License: http://creativecommons.org/licenses/by/4.0/

The gist: We propose Lizard, a linearization framework that transforms pretrained Transformer-based Large Language Models (LLMs) into subquadratic architectures.

Terminology

Abstract

We propose Lizard, a linearization framework that transforms pretrained Transformer-based Large Language Models (LLMs) into subquadratic architectures. Transformers faces severe computational and memory bottlenecks with long sequences due to the quadratic complexity of softmax attention and the growing Key-Value (KV) cache that makes inference memory-bound by context length. Lizard addresses these limitations by introducing a subquadratic attention mechanism that closely approximates softmax attention while preserving model quality. Unlike prior linearization methods constrained by fixed, non-adaptive structures, Lizard augments the architecture with compact, learnable modules that enable adaptive memory control and robust length generalization. Moreover, we introduce a hardwareaware algorithm that solves numerical instability in gated attention to accelerate training. Extensive experiments show that Lizard achieves near-lossless recovery of its teacher model's performance, significantly outperforming previous methods by up to 9.4 - 24.5 points on the 5-shot MMLU benchmark and demonstrating superior associative recall.

Sources

Related papers