Organising Trajectory Evidence for Language-Model Agent Assurance: Fragments, Methods, and the Residual
cs.MA, cs.AI
Submitted: 2026-10-03
Updated: 2026-10-03
License: http://creativecommons.org/licenses/by/4.0/
The gist: Methods for assessing language-model agents include rule checkers over logs, analyses of skill coverage and composition, support checkers, prefix monitors, execution gates, and rare-event estimators.
Abstract
Methods for assessing language-model agents include rule checkers over logs, analyses of skill coverage and composition, support checkers, prefix monitors, execution gates, and rare-event estimators. Each observes a different part of a run and makes a claim of a different strength, and no common account says how these claims combine or what they leave unchecked. We give one, built from a logic, information fragments, and an assurance ledger. Requirements are formulas of a two-tier logic: an outer finite-trace temporal logic over recorded events, and an inner logic of standing over the argument structure that the agent's recorded context supports at a decision. A rule violation and an action taken on withdrawn support are thus formulas of one language. The information available to an assessor, to the agent, and to an execution gate defines fragments of that language; each checking method decides one fragment and returns a typed claim: exact, on a named test suite, a risk bound, or descriptive. The ledger merges the evidence and classifies every obligation as established, addressed but not established, or unaddressed. On 456 released τ squared-bench telecom trajectories, the benchmark oracle flags 231 runs; adding a rule checker, a support stand-in, and a prefix-monitor baseline raises the union to 391, 401, and 407, each contributing flags the others miss, and a gate makes one prohibition exact on a gated deployment. The 49 unflagged runs and the unmet or unaddressed obligations form the residual. Discovery makes unknown requirements explicit, and new or stronger checkers then reduce it.
Sources
- $\tau^2$-Bench: Evaluating Conversational Agents in a Dual-Control Environment
- Grounded Continuation: A Linear-Time Runtime Verifier for LLM Conversations
- SkillFuzz: Fuzzing Skill Composition for Implicit Intents Discovery in Open Skill Marketplaces
- ToolPrivacyBench: Benchmarking Purpose-Bound Privacy in Tool-Using LLM Agents
- PrefixGuard: Online Failure Warning and Trace-Grounded Diagnosis for LLM Agents
- Runtime Compliance Verification for AI Agents
- Skill Coverage: A Test Adequacy Metric for Agent Skills
- AgentSpec: Customizable Runtime Enforcement for Safe and Reliable LLM Agents
- AgentArmor: Enforcing Program Analysis on Agent Runtime Trace to Defend Against Prompt Injection
- SCARCE: Scalable Cascade Analysis for Rare-event Characterisation via Embeddings
Related papers
- Highway Congestion Reduction through Reinforcement Learning Based Eulerian Headway Control
- ORACLE-SWE: Quantifying the Contribution of Critical Information Signals on SWE Agents
- You Only Align Once: Propagating Cooperative Behaviors in Multi-Agent Systems through Seed Agents
- Deny Without Disabling: Authorization-Paired Evaluation and Control for Multi-Agent Systems
- MA-SAPO: Multi-Agent Reasoning for Score-Aware Prompt Optimization
- PeroMAS: A Multi-agent System of Perovskite Material Discovery