Activity Frames: Deterministic Screen-Activity Compilation for Agent Memory and Replay

The paper addresses a fundamental gap in LLM agent memory: computer-use agents pay full frontier inference to rederive routines their user has already performed, because an agent's memory today records what the user said, not what the user did. While capture infrastructure is now commodity—Microsoft Recall, OpenAI's Chronicle, and automated time trackers all record screen activity—consumption is not. A day of event-driven capture on the author's corpus yields roughly two thousand snapshot rows, but raw search costs 126,812 tokens for a single day, while LLM summarization compresses well but imports the weaknesses of its summarizer: per-run cost, non-determinism, degraded reliability over long inputs, and the possibility of inventing activity that never happened. Neither yields memory an agent can cache, audit, or trust.

The paper proposes the missing middle: deterministic compilation—a compiler that turns snapshot streams into activity frames: bounded episodes with an application and site, a start and end, dwell-based active time, typed page references, input volume, and evidence pointers back to the raw rows. Critically, no model participates. The same database and window always produce the same document, so episodic memory becomes free to rebuild, safe to cache, and mechanically auditable.

Four design principles fix the schema: Measured, not guessed (every field in a standard document is derivable by deterministic code from capture data; there are no intent labels, because code cannot observe intent); Reproducible; Evidenced (every frame carries pointers to the raw rows it was compiled from); and Honest about absence (periods without capture are reported as gaps, and every document carries a blind spots list).

The two-tier architecture separates measured fact from inference. Tier 1 is produced entirely by deterministic code. Tier 2 is an optional extension where inferred labels must be namespaced, confidence-tagged, and linked to the measured evidence that supports them—a consumer can always strip the inferred block and be left with a purely measured document. The schema also mandates a privacy rule: input volume is part of the document, but input content (typed text) must be excluded by default and included only on an explicit operator opt-in.

Three constants govern segmentation, chosen from the capture cadence rather than tuned on outcomes:

• Dwell crediting: a frame contributes min(Δt, 90 s) of active time; the cap bounds the credit a single frame can earn when capture stalls or the display sleeps.

• Session gap: a gap above 300 s closes the current frame and becomes a candidate coverage gap.

• Flicker merge: the pattern A → B → A where B lasts ≤20 s collapses into a single A frame, with B not discarded but recorded on the merged frame as an interruption with its measured seconds, so nothing is hidden and nothing is double-counted.

Enrichment repairs three reliability defects by code: stale event attribution (re-attributed via binary search to the temporally nearest snapshot), anonymous clicks (resolved against the element tree via exact containment, then ±40 px tolerance, then zone fallback), and keyboard-layout mismatch (via an explicit operator-supplied translation map, never inferred). Entity typing resolves URLs through layered parsers (20+ bespoke site parsers, generic search-parameter detector, subdomain-path heuristic, and a total fallback) so typing never loses data.

An open MIT-licensed Python package with three parts: a provisioned local capture engine (on-device SQLite, audio off by default), a zero-dependency compiler with JSON/YAML/Markdown/compact plaintext emitters, and a hand-rolled MCP server exposing six tools: get context, get activity, get day summary, get patterns, get communications, and get steps.

On the author's live single-user corpus (61 calendar days, 46 active days, 109,735 snapshot rows, 214,360 input events, 8.4M element-tree rows, 54 applications):

• Token cost: raw rows cost 126,812 tokens/day; the schema-v1 document costs 34,815 tokens (3.6× reduction); the compact context block costs 1,469 tokens, an 86× reduction—small enough to include in every system prompt.

• Downstream QA: On 8 days with 64 questions graded against an independent SQL oracle, activity frames achieve 98.4% accuracy at both the Sonnet 4.5 and Opus 4.5 tiers, versus 82.1%/91.1% for raw rows and 66.1%/80.4% for LLM summaries at the two tiers. The block lets the cheaper model answer as well as the expensive one. Hallucination rate was 0% for all representations on absent-fact probes. The summary failed on magnitudes: at the Sonnet tier it answers time questions at 7.1% accuracy and mis-states the dominant application's active minutes by 135.7% on average, against 7.3% for activity frames. A concrete example: a Sonnet summary named Cursor the primary application at ∼7 hours when the measured record showed Chrome first at 161.6 dwell minutes and Cursor second at 143.9, fabricating a nocturnal session. The summary was non-deterministic (three regenerations produced three distinct texts every day), while the activity-frames block was byte-identical.

• Latency: full-day compilation takes a median of 68 ms—episodic memory at this cost can simply be rebuilt on every query.

• Entity typing: 81.3% of 5,120 distinct URLs receive non-generic typed references spanning 46 kinds.

• Fragmentation: 17,514 frames with median active duration 0.5 min; after excluding single-snapshot transits (52% of sub-minute frames) and dual-monitor effects, the median rises to 0.9 min, consistent with what interruption science has measured.

The compiler doubles as a demand-side cost instrument measuring parameters agent-cost models assume but had not measured. R = C agent(k) / C replay(k), where the numerator is a modeled upper bound of what a memoryless screenshot-driven agent spends per step (screenshot priced at the wh/750 image-token rule plus 350-token context read plus 180-token reasoning write, ≈2,500 tokens/step at 1512×982), and the denominator is measured via deterministic compilation:

• R inject (operational): against the guarded skill plan—a per-step script with expected-element, expected-role, and expected-application guards—the median is 60× (IQR 59–62).

• R info (ceiling): against the minimal replay script's bare information content, the median is 343× at action granularity (198× at URL granularity), robust to routine length (301× at k=3–4 to 405× at k≥16) and screen resolution (259× conservative to 425× retina).

A three-arm modeled comparison of the 20 most frequent routines at Sonnet-class list rates: arm A (memoryless re-derivation) costs 32.8 Mtok / 125.81; arm B (plan injected in context) saves 83.3% of tokens (20.96); arm C (deterministic local replay) saves 40.8% (74.46). A live proof-of-concept executed a compiled two-step routine in an authenticated browser session at zero model tokens, with the fail-safe working as intended on a wrong page. The honest all-fleet ceiling is h(1 − 1/R info) ≈ 9.0% in-sample.

Measured at three levels, explicitly distinguishing the honest figure from inflated ones: h raw = 83.1% (dominated by generic micro-structure of input with no reusable work), h specific = 9.0% at action granularity (13.1% at URL granularity, restricted to routines with ≥2 named targets recurring ≥3 times), and a temporal holdout gives 7.7% out-of-sample. The paper explicitly rejects Web-era revisitation constants of 40–58%: A page revisit is not a delegable task... importing that number would overstate h by roughly an order of magnitude.

A certification harness verifies byte-identical output, rebuild-equals-incremental, and O(Δ) compile cost (median 220.9 ms full-day). The pipeline is fully local, the compiler opens the database read-only, and compilation involves no model of any kind. Trust properties: evidence pointers make episodes mechanically auditable against poisoning attacks, and the measured/inferred boundary prevents compromised tier-2 tools from inject[ing] interpretation disguised as observation—though provenance begins at the database, so a compromised capture engine can fabricate raw rows.

Single-user corpus; the numerator and three-arm dollars are modeled rather than billed; screen presence is an imperfect attention proxy (heartbeat runs reach 42 minutes); dual monitors double-count credit on 27% of captured minutes; capture has operating costs (9.5 GB database, continuous OCR duty cycle); entity typing at 82% propagates into the match-precision q; OCR is a model, so determinism holds forward, not back to pixels—scoped to the compile path over stored capture, never to perception.

The paper claims: a two-tier schema separating measured fact from tagged inference; deterministic compilation rules; an open reference implementation with MCP tools; empirical characterization on a 61-day corpus; a downstream QA benchmark showing the compiled block lets a mid-tier model match a frontier one; and a demand-side cost instrument producing first values of R (60–343×) and h (9.0% in-sample, 7.7% out-of-sample), with schema, compiler, and evaluation harness all open.

1. Deterministic episodic memory that replaces LLM summarization of screen capture. AI systems get a compiler that turns raw snapshot streams into activity frames — bounded episodes with application/site, start/end, dwell-based active time, typed page references, input volume, and evidence row pointers — with no model in the loop. The improved system can rebuild its entire episodic memory in a median 68 ms, cache it safely (byte-identical output on every rebuild), audit any frame back to its raw rows, and answer memory queries with zero hallucination on absent-fact probes — unlike current systems that pay 126,812 tokens/day for raw search or accept non-deterministic, inventable LLM summaries.

2. Always-on daily context at negligible cost. A full day compiles to a 1,469-token compact context block (86× smaller than raw rows), small enough to inject into every system prompt. The improved agent always knows the user's active applications, dwell times, sites, communication targets, and capture gaps across recent days; with that block, a mid-tier model answers 98.4% of activity questions correctly — matching a frontier model — versus 82.1% for raw rows and 66.1% for LLM summaries at the same tier. This lets systems downgrade memory-query workloads to cheaper models without accuracy loss.

3. A measured/inferred two-tier memory architecture. The system separates deterministic measured facts (Tier 1) from any model-produced interpretation (Tier 2), which must be namespaced, confidence-tagged, and linked to the evidence supporting it. The improved system can strip all inference and retain pure measurement at any time; compromised inference tools cannot inject interpretation disguised as observation; and memory stays mechanically auditable against poisoning attacks. This is a general template: never let inference contaminate the ground-truth layer of agent memory.

4. Honest recurrence measurement for capacity planning. The system measures its own recurrence h at 9.0% in-sample / 7.7% out-of-sample (action granularity) instead of importing inflated Web-era constants (40–58%). It distinguishes generic micro-structure of input from genuinely delegable routines, so fleets budget replay infrastructure for the 9% of activity that is actually reusable, and avoid over-building for page revisits that are not delegable tasks. Fleet-level savings forecasts become honest: ceiling h(1 − 1/R info) ≈ 9.0% in-sample.

5. Compiled routine replay with fail-safes, at zero model tokens. By computing the routine overhead ratio R (median 60× operational, 343× informational ceiling), the system knows exactly how much inference it wastes re-deriving routines. The improved agent compiles recurring routines (≥3 recurrences, ≥2 named targets) into guarded scripts — expected element, expected role, expected application guards — and replays them locally at zero model tokens; when a

Computer-use agents pay full frontier inference to re-derive routines their user has already performed, because an agent's memory today records what the user said, not what the user did. We compile passively captured screen activity into agent memory with a deterministic, zero-model pipeline: it segments a local capture stream into typed activity frames, bounded episodes carrying application, site, timing, input volume, and evidence pointers back to the raw rows, with no model in the loop, so the output is byte-identical, cacheable, and mechanically auditable. On one professional's single-user corpus of 128,756 frames over 51 active days, the compiler reduces a day of raw capture to a prompt-ready context block 86x smaller in 68 ms, and an agent reading that block answers questions about the day at 98.4% accuracy (Wilson 95% CI 91.7-99.7%) against an independent oracle, versus 66-80% for an LLM summary of the same capture, a mid-tier model reading the block matching a frontier one. The same compiler doubles as a demand-side cost instrument. Read off passive, pre-delegation human activity rather than agent rollouts, it supplies two parameters that agent-cost models assume but, to our knowledge, have not measured: the Routine Overhead Ratio R and the routine recurrence h. We report first values of R, a modeled upper bound, at 60-343x, and a delegable recurrence of 9.0% in-sample and 7.7% out-of-sample, for a realistic all-fleet token ceiling near 8%; a compiled routine replays deterministically with the model out of the loop, demonstrated live at zero model tokens on a guard-matched hit. Schema, compiler, and evaluation harness are open.