RIS-Kernel: A Model-Agnostic Architecture for Long-Context LLM Inference via Sparse Attention

arXiv:2607.21927 · cs.LG · Submitted 2026-07-24 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "RIS-Kernel: A Model-Agnostic Architecture for Long-Context LLM Inference via Sparse Attention".

Jane: The gist The Reduced Interaction Sampling (RIS) inference engine addresses the O(N2) self-attention complexity in large language models by employing model-agnostic sparse stochastic geometry to reduce attention complexity to O(N…

Tom: First, who's behind it and why it matters.

Paper summary: Tom: So we’ve covered the RIS-Kernel paper today. To wrap up, the key thing here is that they developed this model agnostic architecture to bypass the O of n squared scaling bottleneck of self attention.

Jane: It shows that by using sparse stochastic geometry, they can reduce complexity to O of n log n, making long context LLM inference feasible on commodity hardware without relying on expensive GPU clusters for deep document retrieval.

Lu: The title itself really highlights the solution: RIS-Kernel: A Model-Agnostic Architecture for Long-Context LLM Inference via Sparse Attention.

Meng: In simple terms, this means we can run much larger context windows on standard academic hardware without needing to buy specialized hardware just to process the input.

Lalam: It’s about proving that long context LLM inference is feasible on standard academic hardware without GPU acceleration #pg2.

Tom: The implication is that we can start thinking about deploying these powerful models for tasks that require very large amounts of information where the memory constraints used to be a major blocker.

Jane: This paper suggests that the hardware dependency for long context processing can be significantly reduced by using this smart sparse attention technique.

Lu: It’s an important step because it moves the conversation toward making these powerful models more accessible to a wider range of users and researchers.

Conclusion: Tom: So we’ve seen how RIS-Kernel tackles that O of n squared scaling issue in self attention by using sparse geometry to get down to O of n log n complexity, which lets us run long context inference on standard hardware without needing those massive GPU clusters.

Jane: Exactly. The authors are calling this a model agnostic architecture, which is important because it means you don't have to go into the model weights themselves and retrain anything; you just add this engine on top.

Lu: That’s the core mechanism, injecting runtime sparsity through two modes: Stochastic Mode for getting broad coverage and Structural Mode for keeping local relationships intact in a structured way.

Meng: So, in practical terms, it’s like having two different ways to look at the text—one way to see everything globally, and another way to ensure that if you need a specific piece of information nearby, it’s structurally there.

Lalam: It’s about proving that long context LLM inference is feasible on standard academic hardware without GPU acceleration.

Tom: And the results they show are pretty compelling. They validated this on models like Qwen2-one point 5B-Instruct and TinyLlama-one point 1B, which are the ones that can realistically run on commodity hardware without needing a GPU acceleration for inference.

Jane: Which shows that these sub two billion parameter models can actually respond meaningfully to context delivered at inference time, which is crucial for factual accuracy in real applications.

Lu: The title itself, RIS-Kernel: A Model-Agnostic Architecture for Long-Context LLM Inference via Sparse Attention, really highlights how they’ve managed to solve this by focusing on the attention mechanism itself.

Meng: In simple terms, this means we can run much larger context windows on standard academic hardware without needing to buy specialized hardware just to process the input.

Lalam: It’s about proving that long context LLM inference is feasible on standard academic hardware without GPU acceleration.

Tom: The implication here is that we can start thinking about deploying these powerful models for tasks that require very large amounts of information where the memory constraints used to be a major blocker.

Jane: This paper suggests the hardware dependency for long context processing can be significantly reduced by using this smart sparse attention technique.

Lu: It’s an important step because it moves the conversation toward making these powerful models more accessible to a wider range of users and researchers.

Tom: It opens up a whole new avenue for how we deploy these massive language models for complex reasoning tasks that require reading huge documents in one go.

Faculty of Computing, Federal University of Uberlândia (UFU)

cs.LG

Submitted: 2026-07-24

Updated: 2026-10-09

Comments: 20 pages, 9 figures, 5 tables

Code: https://github.com/santosardr/riskernel

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 84/100

The gist: The gist The Reduced Interaction Sampling (RIS) inference engine addresses the O(N2) self-attention complexity in large language models by employing model-agnostic sparse stochastic geometry to

Key concepts

Self-Attention Complexity (O(N^2))
Standard self-attention in LLMs scales quadratically with the sequence length (N), meaning doubling the context size quadruples computation time. This quadratic scaling limits models to roughly 65k tokens and requires massive GPU resources for long inputs, making it impractical for many applications.
Reduced Interaction Sampling (RIS)
RIS is a model-agnostic architecture that introduces sparse stochastic attention geometries directly into the model's runtime process. It works by generating sparse masks in O(N^2) space using lightweight structures to selectively activate only important connections, effectively reducing computational load while maintaining factual accuracy.
Stochastic vs. Structural Modes
These are two sampling strategies within RIS. Stochastic Mode treats the entire sequence as a uniform pool, drawing global neighbors per pivot for broad coverage. Structural Mode partitions the sequence into blocks and fully connects each block as a clique, ensuring local context is preserved even with tight computational budgets.
Hardware Feasibility on Commodity CPU
A key finding is that RIS-Kernel can perform long-context inference on standard academic CPU servers (16–128 GB RAM) without GPU acceleration. This demonstrates that the O(N log N) geometry successfully manages memory and computation constraints, making long context feasible on typical hardware.

Terminology

Summary

The gist The Reduced Interaction Sampling (RIS) inference engine addresses the O(N2) self-attention complexity in large language models by employing model-agnostic sparse stochastic geometry to reduce attention complexity to O(N log N), making long-context LLM inference feasible on commodity hardware without GPU acceleration.

How it works

The RIS architecture injects runtime sparsity directly into unmodified language models through two sampling regimes: Stochastic Mode for global coverage and Structural Mode for local community preservation The paper describes the RIS-Kernel architecture, a systems-level implementation that injects runtime sparsity directly into unmodified language models [6] through two sampling regimes: Stochastic Mode for global coverage and Structural Mode for local community preservation

The core mechanism involves generating sparse geometric masks in O(N 2) space using lightweight boolean structures to avoid out-of-memory faults during allocation The engine utilizes two sampling strategies to build the attention geometry: Stochastic Mode treats the entire sequence as a uniform pool and draws global neighbors per pivot, while Structural Mode partitions the sequence into blocks of size B = min(0.1N, Bmax) and fully connects each block as a clique before adding global redundant edges

Validation and Performance

Validation focuses on Qwen2-1.5B-Instruct [7] and TinyLlama-1.1B [8], defining the most restrictive regime for long-context factual retrieval In controlled evaluations at 32,768 tokens, RIS-Stochastic at 1% density and 70 ensemble seeds achieves 75.00% accuracy, outperforming the native dense baseline (71.88%) This demonstrates that sparse attention acts as a regularizer: low density (1%) over multiple seeds filters out sequence-level noise, whereas higher density (5%) reintroduces distractor noise

Mode Comparison and Utility Frontier

The paper delineates a clear utility frontier where RIS-Structural is the dominant strategy when the budget is extremely tight and proximal anchor recovery is the binding constraint, while RIS-Stochastic is dominant when the budget permits broader coverage The block-clique geometry guarantees that tokens immediately surrounding any ground-truth anchor are always present in the context window, regardless of budget size

Positional Encoding and Scaling

The architecture handles context windows beyond the model’s trained positional limit through dynamic RoPE scaling, which intercepts config.rope scaling or config.rope parameters at load time to compute the factor from the ratio of requested window to native limit YaRN scaling is shown to preserve positional coherence under extrapolation, exceeding linear interpolation in accuracy at 65,536 tokens

Hardware Feasibility

All evaluations run on commodity, unaccelerated CPU servers (16–128 GB of RAM), demonstrating that long-context LLM inference is feasible on standard academic hardware without GPU acceleration The dominant bottleneck on CPU is L3 cache saturation, and the O(N log N) geometry ensures that scaling the window does not multiply wall-clock cost superlinearly

Conclusion

Sparse attention at 1% density, applied to unmodified model weights, recovers the full contextual signal within the native positional limit The stochastic ensemble sidesteps this by rotating the sparse projection across seeds — a token missed in one view is captured in another, with capture probability governed by 1 − (1 − ρsub) k·N The experiments also delineate where the retrieval kernel ends and where positional encoding takes over The full evaluation ran on a desktop workstation and a departmental server, no GPU involved.

Improvements for AI systems

  1. Bold header: Sparse Attention for Long-Context Inference

The RIS-Kernel architecture reduces self-attention complexity from O(N 2) to O(N log N) using sparse stochastic geometry that fits within commodity memory limits. This allows LLM inference on standard academic hardware without GPU acceleration, enabling the processing of 65k-token contexts.

  1. Bold header: Contextual Retrieval Gain Enhancement

The system can achieve significant retrieval gains, with RIS yielding retrieval gains of up to 14.06 percentage points over the zero-context floor (51.56%) at the 65,536-token limit under YaRN scaling. This demonstrates that sparse attention acts as a regularizer: low density (1%) over multiple seeds filters out sequence-level noise.

  1. Bold header: Mode Selection for Budget Optimization

The system can dynamically select between Stochastic and Structural modes based on resource constraints. RIS-Structural is recommended when the budget is extremely tight and proximal anchor recovery is the binding constraint, while RIS-Stochastic is preferred when the budget permits broader coverage and global regularization.

Abstract

Full self-attention in large language models scales as O(N 2), limiting long-context analysis to 65k tokens and requiring costly GPU clusters. The Reduced Interaction Sampling (RIS) inference engine addresses this as a model-agnostic architecture. Without modifying weights, RIS introduces sparse stochastic attention geometries that operate under extreme pruning (up to 99% under 1% density) while preserving factual retrieval. While foundational graph-theoretic RIS formulations establish an asymptotic O(N log N) bound, this work evaluates a practical fixed-density regime on unaccelerated commodity CPU systems. We validate RIS on Qwen2-1.5B-Instruct across two regimes. At 32,768 tokens (where native dense attention serves as the upper bound), RIS-Stochastic at 1% density and 70 ensemble seeds achieves 75.00% accuracy, outperforming the native dense baseline (71.88%), while 5% density at 10 seeds matches it (71.88%), demonstrating that multi-seed sparsity filters sequence-level noise. Crucially, a controlled ablation evaluating a single static mask at matched budget (N=1, d=0.50) yields only 53.12% accuracy, proving that multi-seed topological diversity, rather than raw edge density, drives contextual recovery. Under tight budgets, RIS-Structural reaches 65.62% accuracy at 1% density with 10 seeds (recovering 50.0% of the gap over the 59.38% zero-context floor) and 68.75% (75.0% recovery) with 30 seeds. At 65,536 tokens, where dense attention triggers out-of-memory faults, RIS yields retrieval gains of up to 14.06 percentage points over the zero-context floor (51.56%), confirmed as marginally significant under McNemar's paired test (p = 0.078 < 0.10). All evaluations run on commodity CPU servers (16-128 GB RAM), demonstrating that long-context LLM inference is feasible on standard academic hardware without GPU acceleration.

Sources

Related papers