Runtime Observability for Heterogeneous Attention Memory

The paper is motivated by a production incident: "On 2026-06-25 a silent corruption in a hierarchical cache took down agentic traffic on one of our production hosts. Nothing in the serving stack was in a position to notice: the cache was structurally valid, the model kept generating, and the failure surfaced only as degraded task success far downstream. The authors' core claim is that A compressed or shared attention memory is state that the system trusts without verifying, and the trust is currently blind."

The key insight is that "Contemporary architectures keep their attention memory in at least four different forms: dense per-token key/value tensors; a latent cache decompressed on read; a learned selector that picks a sparse subset of entries; and a recurrent state carried forward rather than stored. They fail differently, they admit different guarantees, and a bound proved for one says nothing about another. The paper notes that The obvious response—'check the KV cache'—no longer names a single object."

Three operators. Any attention memory is written as three operations: mt+1 = Ut(mt, xt) (update/write), It = St(qt, mt) (select, possibly the identity), and yt = Rt(qt, mt, It) (read). Compression or corruption replaces mt by m̂t; the goal is to bound ∥yt − ŷt∥ from locally computable witnesses. The four memory objects are instances: "a dense KV cache appends and selects everything; a latent cache appends a compressed representation and reconstructs on read; a sparse selector adds a non-trivial S; a recurrent state makes U a contraction rather than an append."

Local contracts. Definition 1: A stage exposes (a, b, δ) with eout ≤ a ein + b holding except with probability δ. Series composition is affine: (a2, b2) ◦ (a1, b1) = (a2a1, a2b1 + b2) with Σδi ≤ δreq. The authors stress that What is not elementary in practice is that each a, b and δ must be computable from quantities the serving stack actually has, and that the resulting number must carry an honest strength label.

Three tiers, machine-decided. Every emitted metric is registered as: certified (a sound upper bound exists and is not saturated), partially certified (the bound holds only on a decidable subset), or empirical (an observation with no bound attached). "Composition takes the weakest tier of its stages. This is what makes the distinction operational rather than rhetorical: a chain containing one empirical stage cannot be reported as certified, whatever the accompanying prose says. An unregistered metric is a hard error rather than a silent demotion."

Pooling is shown to be an attenuator, not an error source. Lemma 1 gives two calibers: with ein:= (Σ∥∆j∥2)(1/2), Cauchy–Schwarz gives a = ∥w∥2 ≤ 1; with ein:= max∥∆j∥, convexity gives a = 1 with no assumption. Both calibers must be reported: the l2 form is tighter but presumes an l2 aggregation of the incoming error, and quoting it alone suggests the attenuation is unconditional. The authors note a mistake they made: Placing it inside the baseline chain, as we initially did, yields a bound that is both looser and semantically wrong.

Selection is additive, not amplifying. Lemma 2 (rank-prefix selection): TV(p, p̂) ≤ ½(mout + min) where mout and min are the swapped attention masses, giving (aS, bS) = (1, ½(mout + min)). The selection stage therefore adds a bias equal to half the swapped attention mass and does not amplify upstream error. This reframes a negative result: asking whether the entire top-k set is preserved is the wrong question, because strict set certification is almost never attainable, whereas the swapped mass is small and directly measurable.

Sampling sentinel. With r uniform slots sampled with replacement per round from M slots of which B are corrupted, Pr[missed after n rounds] = qn where q = (1 − B/M) r. Lemma 3 gives detection latency: For target confidence 1 − δ, detection of at least one corrupted slot is guaranteed within n(δ) = ⌈log δ / log q⌉ rounds. The authors emphasize modeling the actual implementation: Sampling without replacement would give the smaller hypergeometric q; modelling the implementation one actually runs is the point. This converts an anecdotal demo into a monitoring guarantee with a tunable budget.

The implementation separates four concerns: "a registry declaring where to inject for each architecture; adapters that extract the tensors and build the validity mask; meters holding the mathematics, which are architecture-independent; and a runtime that keys every accumulator by (owner, layer, path) and always emits a coverage record. Injection is anchored on exact upstream source blocks with an asserted hit count, and reverts surgically so that two adapters may share a file."

Acceptance is a fixed order—coverage, then magnitude, then soundness—and a failure at any step invalidates the numbers that follow. This is justified empirically: "All three checks correspond to failure modes we hit in which the experiment terminated normally and produced plausible output: a probe whose per-layer state overwrote itself, a probe that read uninitialised device memory beyond a kernel's valid range, and a probe attached to a write path the model never takes. Each produced a clean exit and a wrong number."

Coverage. Table 1 summarizes 8 rows / 6 models across five families including Qwen2.5-7B (GQA, KV), Llama-3.1-8B (GQA, KV), DeepSeek-V2-Lite (MLA, LatentKV), GLM-5.2-W4AFP8 (MLA+DSA, LatentKV and SparseSelector), V4-Flash-FP8 (MLA+entry, LatentKV and SparseSelector), and Kimi-Linear-48B (KDA+MLA, RecurrentState). Two models were onboarded without writing any model-specific code.

Per-stage readings. Latent memory: "GLM-5.2's own fp8 storage residual is 0.93% of the latent norm, and our secondary int8 quantisation of the same latents measures 0.551% over all 78 layers. Latent norms are strongly heterogeneous across depth—a factor of 493× between the largest and smallest. Sparse selection: Strict certification of the entire selected set is essentially unattainable: 0.05% on GLM's token-level top-2048. The rank-prefix view is the opposite: top-1 stability is certified for 99.26% of rows on GLM and 99.40% on V4's page-level top-512, with zero flips inside the certified region. Pooling: The normalised concentration at block 4 is 0.1551, i.e. close to uniform. Recurrent state: The per-layer mean contraction is ā ∈ [0.530, 0.936], all below one, but the mean is the wrong statistic: long-run behaviour is governed by Σt log at, and by Jensen's inequality E[log a] ≤ log ā. Measured, E[log a] has median −0.5912 against log ā = −0.2282, so the mean caliber understates the contraction by 2.6×; the corresponding error half-life is 1.17 steps rather than 3.04. No certificate is claimed here because a residual gap remains: between 0.0 and 28.7% of samples per layer have a ≥ 0.999, and our aggregate does not retain channel identity."

Composition failure (type checker rejecting their own chain). The authors report: "Our first composed number added the storage stage's bound to the selection stage's bS by (4) and reported the result. That composition is illegal, and an earlier version of this paper contained it. The storage bound is a relative witness on cache entries—a dimensionless ratio—while bS is total variation on an attention distribution, a probability mass; a2b1 + b2 presumes b1 is already denominated in the output metric of stage two, and here it is not. The fix: each contract declares an input and an output metric, composition is defined only when they coincide, and the same rule is enforced twice—at runtime, where the checker rejects the chain construction, and in Lean, where the ill-typed composition fails elaboration. Neither the tier system nor the numeric claim guard had caught it: both operate on values, and the values were individually correct."

The repaired chain crosses metrics through two proved bridges. The score bridge is Cauchy–Schwarz: ∆(scale · ⟨q, k⟩) ≤ scale · ∥q∥ · ∥∆k∥, with runtime coefficient measured as scale · max∥q∥ = 1.0034 across V4-Flash layers. The distribution bridge: a score perturbation of at most ε moves the softmax distribution by at most TV ≤ ½(e 2ε − 1).

A second mislabel was caught in review: our selection bound bS is computed from the softmax of indexer logits, so its unit is selector-score mass, not attention mass; we had typed it as attention-TV by fiat. The two quantities are therefore reported separately and never summed. The remaining window-storage term gives a per-layer attention-TV bound of 0.786 under the all-rank worst caliber... Summed over all 43 layers the budget is 36.3—vacuous, and reported as such. Both scope restrictions were discharged by measurement rather than by more formalism: with every term witnessed on the 21 sparse-selection layers the full chain gives a worst-caliber per-layer TV bound of 1.473 (median), and the selector-to-attention gap is bridged by "a paired measurement—the ratio of attention mass to selector-score mass is 0.87 (median; converted with the tail value 2.18)—so the selection term crosses into attention units at tier empirical, and the composed tier drops with it, automatically. The authors note: These numbers are larger than the retracted ones because they pay the full price of soundness... A bound that got smaller by adding incompatible units was not a bound."

Request-level risk ledger. Three rules: "empirical objects never enter the certified ledger (they either fall back or degrade the request's verdict); deterministic certificates... enter at δ = 0; and probabilistic certificates draw from a telescoping budget δi = δreq/((i+1)(i+2)), Σδi < δreq, so the guarantee holds at every request length without knowing the length in advance. The budget rules are machine-checked theorems; the exclusion rule is not a theorem but an API discipline. Early replay attempts failed because of granularity: Our first replay certified whole decode steps against the step's worst sampled entry; one large-witness entry then forced the entire step to fall back, and coverage was 0% at every useful threshold. Local fallback must be local. With entry-level certification, replaying 12.4M entry reads across the 21 sparse-selection layers gives the trade-off curve—80.1% compressed-entry retention at a vacuous per-step bound of 1.54, or TV ≤ 0.86 at 29.3% retention. They say we say retention rather than certified coverage: quoting the retained fraction alone would hide that its per-step bound is vacuous."

The online shadow ledger closes four of the six gaps between replay and execution: 198,163,408 entry events over 24 serial requests with median per-request retention 0.334. The write-time experiment uses FP8 mantissa masking (precision coarsening, not physical compression), with "pre-authorization semantics: a prior radius authorizes an entry before any random bits are drawn, a write-side audit counts realized violations (0 of 147,208 authorized entries), and the price of authorizing before the draw is quantified rather than hidden (6.2× fewer entries than post-hoc gating accepts). Under eight-way concurrency, every account independently satisfies its budget (maximum spend Σδ = 0.009984 ≤ δreq) with the owner ledger reporting zero foreign-owned reads across 8,228,864 checks. The concurrent run flagged 3.4% foreign reads and traced them to a namespace collision between the two compressed-pool families—invisible under serial traffic."

Four runtime invariants make shared-read semantics fail-closed: write-audit before any shared read, explicit slot state, on-the-spot byte restoration (admission is probabilistic, enforcement is deterministic), and once-per-write budget spend (shared read sound). On output quality, the control is reported first: "greedy decoding across two independent server restarts with no intervention already disagrees on 2 of 8 diverse prompts. Against that stability floor, the policy and dithered arms' output differences from baseline are at or near the floor at n=8 and cannot be resolved."

Upgrading the certified object. The witness being the binding constraint, we changed the certified object itself: from a worst-case norm to the exact conditional distribution of stochastic rounding. The centred bound TV ≤ tanh(osc/4) moves non-vacuous layers from 0/43 to 43/43 on this object. The adjudicator's soundness self-check caught three of our own bugs: "a two-sided Chernoff combined with min instead of max; a deterministic shift from top-of-range clamping that is not random at all; and a zero-weight branch producing log 0 that a clamp silently turned into fake perfection. One gate failed and is honored: no family reaches τ=0.2 retention at two-bit coarsening, so the heavy fused-kernel stage is skipped by the pre-registered rule. The finite-product radius raises the pre-authorization rate at the same threshold from 7.8% to 67.5% (8.7×), with 0 of 1,276,631 authorized entries violating their radius. An anytime-valid e-process runs on the same object as the ledger": Over 120,661 authorized-entry factors spanning all requests, the global process never approaches its crossing threshold: sup log Mt = −0.82 ≪ ln(1/δe) = 4.61.

Case study and discrimination campaign. A low-bit format study falsified two of our own design assumptions: entry slots evolve progressively, so write-time packing reads a non-final state and the apparent Python read path is not the production consumer. A 95-point retrieval collapse was "discriminated down to generation staleness—slot reuse with no invalidation call anywhere in the codebase—rather than quantization (the quantizer's intrinsic relative error is 0.0245, unreachable from the observed readings), and unconditional write-site invalidation restored accuracy with a two-sided causal confirmation (1.000 vs. 0.200 under 4-way concurrency, six machine-checked criteria). The observation site was shown to be part of a metric's type: The write-site shadow check—mathematically identical to the decode-side one—had zero discriminative power for this failure while the same metric at the consumption point separates the arms by 36×: 0.0246... against 0.882."

The machine-adjudicated discrimination campaign produced the paper's central structural finding: every observed failure occurs in an eviction or slot-reuse regime, while eviction alone does not force failure (one 91-eviction control run scores 1.000). The zero-eviction, identity-isolated configuration is exact across three independent twelve-document rounds, and it is the only configuration this campaign certifies. The exclusion list is long: "beyond the write chain and the persistent stores, a materialized reference reader... reproduces the same 0.500, so the production kernel's inline unpack and the reader mathematics are exonerated as well. What remains is a stable structural association... whose microscopic mediator is open, with the remaining candidate surface in selection metadata, index semantics, or their timing state. A per-pool dissociation pins the pool: evicting only the 128× ring is exact (reproduced three times), evicting only the 4× ring degrades to 0.667. The intervention of folding non-resident reads to the kernel's skip sentinel restores the eviction microcosm from 0.500 to 1.000 against a knob-isolated control with matched eviction and call counts but is a mitigation, not a repair — when eviction overlaps the prefill window the same truncation costs the needle's context."

Path-contract evaluation. Six serving paths are evaluated against the packed stage with a five-level taxonomy: pass / fail-method / blocked-upstream / partial / not-measured (Table 2). Key lessons: "CUDA-graph serving is exact at the eviction-free caliber and caliber-limited beyond it (0.75 at twelve-document concurrency); radix prefix caching fails with the m…

1. Always-on faithfulness monitoring for heterogeneous attention memory.

Current systems trust compressed or shared attention state blindly — the motivating incident: a structurally valid cache that silently corrupted, the model kept generating, and the failure surfaced only as degraded task success downstream. This framework replaces blind trust with a per-stage contract eout ≤ a·ein + b, held with probability δ, over three operators (update, select, read), instantiated for dense KV caches, latent caches, sparse selectors, and recurrent states.

→ The improved system continuously certifies whether the retained attention memory is still faithful, per layer, per step, per request — and flags drift before it compounds into task failure, instead of discovering degradation far downstream.

2. Metric-type discipline for composed guarantees.

The system's own first composed bound was illegal: it added a dimensionless storage residual to a probability-mass selection bias. A chain that mixes incompatible units is rejected at composition time.

→ The improved serving stack can never report an end-to-end bound that is arithmetically computed but semantically meaningless. It enforces this twice — at runtime and in a proof assistant — so ill-typed guarantee chains fail construction.

3. Machine-decided tiering: certified / partially certified / empirical.

Every emitted metric carries an honest strength label; an unregistered metric is a hard error, and composition takes the weakest tier. The tier system is what forced the authors' own true residual to be rejected as unavailable at runtime.

→ The improved system cannot accidentally present empirical observations as certified guarantees, no matter how convincing the accompanying prose or how plausible the number. When a bridge is only measured, the composed tier drops to empirical automatically.

4. Entry-level local fallback instead of whole-step fallback.

Early replay certified whole decode steps against each step's worst entry; one large-witness entry forced the entire step to fall back, yielding 0% coverage at useful thresholds. Entry-level certification rescued the trade-off: 80.1% compressed-entry retention at a vacuous per-step bound, or TV ≤ 0.86 at 29.3% retention — reported honestly as retention, not certified coverage.

→ The improved system degrades gracefully: a single bad entry costs that entry, not the step; and it reports both the retained fraction and the per-step bound so operators know exactly what they are buying.

5. Root-cause discrimination that separates corruption mechanisms.

A 95-point retrieval collapse was discriminated down to generation staleness (slot reuse with no invalidation anywhere in the codebase) rather than quantization — the quantizer's intrinsic error was 0.0245, unreachable from observed readings. The write-site shadow check (mathematically identical to the decode-side check) had zero discriminative power, while the same metric at the consumption point separated arms by 36× (0.0246 vs. 0.882).

→ The improved system attributes quality regressions to the correct mechanism (stale slots, not quantization) and knows that the observation site is part of the metric's type — a monitor at the wrong site is not a weaker monitor, it is a silent non-monitor.

6. Structural failure-regime certification.

Machine-adjudicated discrimination produced a sharp boundary: every observed failure occurred in an eviction or slot-reuse regime; zero-eviction, identity-isolated configuration was exact across independent rounds; eviction alone does not force failure (one 91-eviction control scored 1.000). A per-pool dissociation pinned the culprit pool (128× ring), and folding non-resident reads to the skip sentinel restored the microcosm from 0.500 to 1.000.

→ The improved system reports which configurations it certifies and which it does not, with the microscopic mediator honestly marked open — rather than presenting a global it works from favorable conditions.

7. Fail-closed concurrency semantics for shared reads.

Four runtime invariants — write-audit before shared read, explicit slot state, deterministic on-the-spot byte restoration, once-per-write budget spend — made shared-read serving enforce per-account budgets. Under 8-way concurrency, every account satisfied its budget (max spend Σδ = 0.009984 ≤ δreq), and the system caught a 3.4% foreign-read rate traced to a namespace collision between compressed-pool families that was invisible under serial traffic.

→ The improved system serves concurrent requests over shared compressed memory with per-account probabilistic budgets that are enforced deterministically, and detects cross-owner contamination that serial testing cannot expose.

8. Pre-authorization write-side auditing.

Write-time precision coarsening (FP8 mantissa masking) uses pre-authorization: a radius authorizes an entry before random bits are drawn, a write-side audit counts realized violations (0 of 147,208 authorized entries), and the price of authorizing blind is quantified, not hidden (6.2× fewer entries than post-hoc gating).

→ The improved system can compress memory at write time with a guarantee about the authorized entry's fidelity, and it tells you the throughput price of that guarantee explicitly.

9. Upgrading the certified object to beat vacuity.

Switching the certified object from a worst-case norm to the exact conditional distribution of stochastic rounding, with the bound TV ≤ tanh(osc/4), moved non-vacuous layers from 0/43 to 43/43; the finite-product radius raised pre-authorization from 7.8% to 67.5% (8.7×) at the same threshold. The adjudicator's self-check caught three internal bugs, including a fake-perfection bug from a clamp silencing log 0.

→ The improved system chooses the object of certification to match the actual error process — making guarantees non-vacuous where worst-case thinking said impossible — and its own validation machinery rejects fabricated perfection.

10. Graph-aware instrumentation that survives CUDA-graph capture.

Four silent-corruption trap classes were catalogued: accumulators must be device-resident tensors, sampling must avoid RNG, flushing must be deferred outside the capture window, and persistent accumulators must not be allocated during capture. A probe not in the graph is free — and a run that passed acceptance on leftover samples was exactly that trap. The fused Triton meter costs −0.94% throughput and −0.14% TTFT P99 at 24.2× lower per-invocation cost.

→ The improved system maintains live observability inside graph capture, with counters that are actually executed, not leftovers from before capture — so coverage over the run's lifetime is not mistaken for liveness during the run.

11. Accurate sentinel bounds by modeling the actual implementation.

Sampling with replacement (the real implementation) gives detection latency q = (1 − B/M) r; modeling it correctly gives: 99% confidence for one corrupted slot in 148/74/37/19 rounds at r = 32/64/128/256. The earlier hypergeometric analysis was optimistic about the implementation it described.

→ The improved system converts anecdotal spot-checks into monitoring guarantees with tunable budgets, and its latency claims match the code actually running — not a friendlier abstraction of it.

12. Honest path-level accounting.

Six serving paths were scored against a five-level taxonomy (pass / fail-method / blocked-upstream / partial / not-measured): CUDA-graph serving exact only at eviction-free caliber; radix prefix caching reproducibly failed (0.667 vs. 1.000); prefill/decode disaggregation returned 0.000 with zero transport or server errors — structurally green and semantically wrong. Overhead was reported at three calibers with tail latency as the always-on criterion and per-setting noise floors.

→ The improved system tells operators which serving path is certified under which conditions and refuses to let no errors logged stand in for output correct — closing exactly the gap a runtime quality oracle exists to close.

Net effect: the improved AI serving system observes every declarable attention-memory object against honest, tiered, type-checked contracts; detects and attributes silent corruption and staleness early; certifies only what it can prove, says vacuous when vacuous; fails closed under concurrency and graph capture; and produces request-level risk ledgers that operators can actually act on — instead of blind trust in a structurally valid cache.