PRISM: Recovering Instruction Sets from Language Model Activations
cs.AI, cs.LG
Submitted: 2026-06-08
Updated: 2026-09-27
Comments: Accepted to EMNLP 2026 (Main Conference). Code: https://github.com/Offensive-AI-Lab/prism
Code: https://github.com/f/prompts.chat
License: http://creativecommons.org/licenses/by/4.0/
The gist: As LLMs are deployed as agents, reliable monitoring requires knowing not only what they output, but which instructions are steering their behavior.
Terminology
Abstract
As LLMs are deployed as agents, reliable monitoring requires knowing not only what they output, but which instructions are steering their behavior. This is difficult when models infer unintended subgoals, follow contextual cues, or are influenced by prompt injections and hidden objectives. While activation-to-language methods suggest that hidden states can reveal natural-language information, existing approaches are not designed to recover the full set of simultaneous instructions, constraints, prohibitions, and subgoals present in agentic settings. As a first step toward this monitoring goal, we formalize instruction set retrieval and introduce PRISM, an activation-conditioned interpreter that decodes hidden states from a frozen target model into a faithful bullet list of instructions. Unlike prior activation-to-language methods, PRISM is trained to recover instruction sets directly, using judge-guided GRPO to reward covered instructions and penalize unsupported ones. Across benign, constrained, prompt-injection, and hidden-objective settings, PRISM outperforms activation-to-language baselines, especially on security-relevant objectives.
Sources
- Obfuscated Activations Bypass LLM Latent-Space Defenses
- Probing Classifiers: Promises, Shortcomings, and Advances
- Discovering Latent Knowledge in Language Models Without Supervision
- Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection
- RL-Obfuscation: Can Language Models Learn to Evade Latent-Space Monitors?
- Designing and Interpreting Probes with Control Tasks
- WildTeaming at Scale: From In-the-Wild Jailbreaks to (Adversarially) Safer Language Models
- Activation Oracles: Training and Evaluating LLMs as General-Purpose Activation Explainers
- Large Language Models Often Know When They Are Being Evaluated
- LatentQA: Teaching LLMs to Decode Activations Into Natural Language
- Ignore Previous Prompt: Attack Techniques For Language Models
- Generalizing Verifiable Instruction Following
- ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs
- DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
- False Sense of Security: Why Probing-based Malicious Input Detection Fails to Generalize
- Instruction-Following Evaluation for Large Language Models
Related papers
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Agents' Last Exam
- The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
- MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
- Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems