An Exploratory Ablation of a Small MLA--SSM Hybrid Language Model
cs.CL, cs.AI, cs.IR, cs.LG
Submitted: 2026-08-28
Updated: 2026-08-28
Comments: 6 pages, 7 figures. Exploratory single-seed ablation study. Code and replication package available at https://github.com/unseen1980/talh
Code: https://github.com/unseen1980/talh
License: http://creativecommons.org/licenses/by/4.0/
The gist: We report an exploratory, single-seed ablation of TALH (Adaptive Latent Hybrid), a decoder-only language model with parallel Multi-head Latent Attention (MLA) and a custom recurrent state-space (SSM)
Terminology
Abstract
We report an exploratory, single-seed ablation of TALH (Adaptive Latent Hybrid), a decoder-only language model with parallel Multi-head Latent Attention (MLA) and a custom recurrent state-space (SSM) branch. Five variants, spanning 117--217M estimated active parameters per token, are trained from scratch on a FineWeb sample for the same number of optimisation steps and tokens. In this specific setup, removing the SSM branch gives the largest degradation in validation perplexity (MLA-only PPL 315), whereas removing MLA has a much smaller effect (SSM-only PPL 239). A dense-FFN hybrid obtains PPL 231, compared with 240 for the tested top-2 ternary-MoE hybrid, while using 3.87 GB less peak training memory. We also preserve a preliminary Apple M3 timing observation: among the five unoptimised implementations, MLA-only has the flattest measured time-to-first-token curve from 512 to 2,048 prompt tokens, although the dense Transformer is much faster in absolute terms. Because the runs are single-seed, parameter counts are unmatched, the evaluation stream may overlap the training source, and raw repeated timing records are unavailable, these results support implementation-specific hypotheses rather than general conclusions about MLA, SSMs, or mixture-of-experts models.
Sources
- DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model
- Mamba: Linear-Time Sequence Modeling with Selective State Spaces
- Hungry Hungry Hippos: Towards Language Modeling with State Space Models
- Jamba: A Hybrid Transformer-Mamba Language Model
- Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modeling
- Hymba: A Hybrid-head Architecture for Small Language Models
- Zebra-Llama: Towards Extremely Efficient Hybrid Models
- GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints
- BitNet b1.58 2B4T Technical Report
- The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale
- RULER: What's the Real Context Size of Your Long-Context Language Models?
- CoSMoEs: Compact Sparse Mixture of Experts
- Hybrid Architectures for Language Models: Systematic Analysis and Design Insights
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering