Algorithmic Unverifiability of Safety for Fixed and Recursively Self-Improving Systems
cs.LO, cs.AI, cs.CC, cs.CL
Submitted: 2026-06-26
Updated: 2026-10-14
Comments: v3: revised & retitled. Part II covers behavioral/code-reading self-modification, framing safety-generality via restricted expressivity (not stasis); supervisory regress drops Turing-complete assumption. Resource face: coNP-complete verification, PSPACE-complete synthesis over succinct arenas. Part III assumes effectively pointwise properties. 30 pp. Companion: arXiv:2609.11326
License: http://creativecommons.org/licenses/by-nc-nd/4.0/
The gist: We establish mathematical limits of algorithmic safety verification for Turing-complete self-modifying systems, the class in which recursive self-improvement takes place, both for a fixed system and
Terminology
Abstract
We establish mathematical limits of algorithmic safety verification for Turing-complete self-modifying systems, the class in which recursive self-improvement takes place, both for a fixed system and across its own modification. Statically, no verifier is sound, complete and tractable: over unbounded domains by Rice's and Gödel's theorems, over all finite configurations by Trakhtenbrot's theorem, and over succinctly described finite environments because verifying a policy against an adversary is coNP-complete and synthesising one is PSPACE-complete. Dynamically, we model one step of self-modification as a computable transformation of code and ask whether a safety property survives it. If the transformation depends only on behaviour, this is Rice's theorem one level up; if it reads the code, as self-modification does, the question is no longer semantic, yet the same s-m-n reduction works inside a class of behaviourally identical programs and inherits the halting degree. One step is never harder than the property; persistence along the whole trajectory can be Π 0 2-complete. Certification by a total algorithm is possible only for transformations of restricted expressivity, not merely for systems that stop changing. No tower of supervisors helps, and every total supervisor errs on an undecidable set of systems. For effectively pointwise properties, every faithful bounded scheme that certifies on finite behavioural evidence admits evolution traces certified at every stage while the property is violated. What survives is exact: a monitor that raises an alarm on violation semidecides it, and comparison against a frozen reference keeps the full theory.
Sources
- No Certificate for Alignment: Two Independent Impossibilities and the Pareto Frontier of Achievable Safety Guarantees
- Concrete Problems in AI Safety
- Supervising strong learners by amplifying weak experts
- AI safety via debate
- Scalable agent alignment via reward modeling: a research direction
Related papers
- Formalizing Flag Algebras in Lean
- An Information-Flow Perspective on Explainability Requirements: Specification and Verification
- A programming language combining quantum and classical control
- Causal Past Logic for Runtime Verification of Distributed LLM Agent Workflows
- ADVENT: LLM-Driven Automatic Predicate Invention for ILP
- Encoder-Decoder Transformers: Logical Characterizations and Periodicity