GatedV: Self-Gated Values for Transformer Language Models
cs.CL, cs.LG
Submitted: 2026-11-01
Updated: 2026-11-03
Comments: 12 pages, 3 figures. Added Hongyu Zhou as a co-author. Code and experimental results are available at https://github.com/yuliming1996/GatedV
Code: https://github.com/yuliming1996/GatedV
License: http://creativecommons.org/licenses/by/4.0/
The gist: Standard attention forms query (Q), key (K), and value (V) matrices through learned linear projections of input hidden states.
Terminology
Abstract
Standard attention forms query (Q), key (K), and value (V) matrices through learned linear projections of input hidden states. Architectures such as GAU and Mega incorporate nonlinear transformations into attention. In this paper, we propose GatedV, a simple attention variant that applies SiLU only to V before aggregation without introducing additional trainable parameters. We evaluate GatedV against a standard decoder-only Transformer using matched parameter counts, paired initializations, and equal training-token budgets. At 2.33M parameters, GatedV reduces mean held-out perplexity from 59.1 to 57.9 (2.0%), with improvements across all five paired seeds. At 26.16M parameters, following an equal-budget learning-rate search, mean held-out perplexity falls from 20.2 to 19.7 (2.4%), with improvements across all three paired seeds. Small-model ablations show that the tested affine rescaling of values does not fully reproduce GatedV's gains. A short interleaved benchmark on NVIDIA DGX Spark measures approximately 0.7% higher full-context forward latency for the larger model, based on synchronized wall-clock timings. These findings suggest that value-only self-gating can improve language modeling performance without increasing parameter count or redesigning the attention-weighting mechanism.
Sources
- Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning
- Pointer Sentinel Mixture Models
- Searching for Activation Functions
- RoFormer: Enhanced Transformer with Rotary Position Embedding
- Neural Attention: Enhancing QKV Calculation in Self-Attention Mechanism with Neural Networks
Related papers
- ALARM: Audio-Language Alignment for Reasoning Models
- ThinkSum: Probabilistic reasoning over sets using large language models
- Coherence boosting: When your pretrained language model is not paying enough attention
- From Facts to Insights: A Persona-Driven Dual Memory Framework and Dataset for Role-Playing Agents
- Improving General Role-Playing Agents via Psychology-Grounded Reasoning and Role-Aware Policy Optimization
- The Art of Asking: Multilingual Prompt Optimization for Synthetic Data