Elicitation without Backpropagation: Steering Model Behavior by Optimizing the Latent Posterior
Garrett Baker, Vinayak Pathak, Daniel Murfet, Susan Wei
cs.LG, stat.ML
Submitted: 2026-07-21
Code: https://github.com/timaeus-research/elicitation
License: http://creativecommons.org/licenses/by/4.0/
The gist: In the latent posterior model of transformer behavior, the next-token distribution arises from a posterior over latent predictive models conditioned on the context, mixed to generate continuations.
Terminology
Abstract
In the latent posterior model of transformer behavior, the next-token distribution arises from a posterior over latent predictive models conditioned on the context, mixed to generate continuations. We exploit this model in settings where it is exact, namely Bayes-filtered transformers (BFTs) meta-learned on sequences from a hierarchical prior, to introduce Posterior Prefix Tuning (PPT), a new method for eliciting behavior from a transformer: given a utility function on continuations, find a prompt under which the transformer generates continuations of high expected utility. For a BFT, the elicitation objective factors through the latent posterior, and the gradient of this objective can be estimated from samples of the prior alone. PPT optimizes the parameters of a distribution over hard prompts: it draws prior samples once from the BFT via predictive Monte Carlo (PMC), then estimates the gradient by importance sampling against them. The optimization performs no transformer forward passes and no backpropagation through the transformer, and the prior samples are utility-independent, so a single set of samples drives elicitation against any number of utilities at negligible marginal cost. We validate PPT on Beta--Bernoulli and reinforced urn BFTs across three utility families (reverse cross-entropy, frequency matching, Dyck validity).
Sources
- Uncertainty Decomposition for Bayes-Filtered Transformers via Bayesian Predictive Inference
- Meta-learning of Sequential Strategies
- Next-token pretraining implies in-context learning
- Scalable and Transferable Black-Box Jailbreaks for Language Models via Persona Modulation
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks