A Parametric Linear Conic Projection for Structured Sparse Attention
cs.AI
Submitted: 2026-11-03
Updated: 2026-11-03
Code: https://github.com/riohib/gsp-for-deeplearning
License: http://creativecommons.org/licenses/by-nc-sa/4.0/
The gist: In this paper, we introduce the Cone Alignment Index (CAI), a scale-invariant sparsity measure associated with a Lorentz-cone geometry.
Terminology
Abstract
In this paper, we introduce the Cone Alignment Index (CAI), a scale-invariant sparsity measure associated with a Lorentz-cone geometry. We provide a probabilistic analysis of the distribution of the CAI score, including a general theorem and explicit corollaries for Gaussian and uniform distributions. As a consequence, we show that, in the high-sparsity regime, the probability that the samples lie within the convex cone tends to one. Building upon this formulation, we derive two complementary projections: the Parametric Linear Conic Projection (PLCP), an efficient closed-form approximation, and the Parametric Euclidean Conic Projection (PECP), the exact Euclidean projection onto the cone boundary. Our analysis establishes several fundamental theoretical properties. We prove that PLCP and PECP are collinear, differ only by a positive radial scaling factor, and induce exactly the same sparsity threshold, thereby preserving an identical active support. Furthermore, we derive a support-identification theorem showing that, for any prescribed sparsity level l<n, the number of coefficients discarded at each iteration is theoretically controlled. Building on this theoretical foundation, we develop a bilevel PLCP framework for structured matrix sparsification. The proposed method naturally generates hardware-friendly structured sparsity patterns, including row-wise, column-wise, and diagonal masks, making it particularly well suited for accelerating Transformer attention mechanisms.
Sources
- Comparing Measures of Sparsity
- Mixtral of Experts
- Transformed $\ell_1$ Regularization for Learning Sparse Deep Neural Networks
Related papers
- Multi-Objective Structured Pruning of LLMs for Latency and Model Size Optimization
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Embeddings for Preferences, Not Semantics
- Can We Trust Item Response Theory for AI Evaluation?
- Diagnosing Task Insensitivity in Language Agents