When Do Prompt-Side Agent Playbooks Transfer? Accuracy, Cost, and Runtime Shift in Agent Deployment

The paper studies frozen prompt-side playbook transfer: compact prompt-side operating procedures distilled from prior runs, validated on the source side, and then transferred to new deployment settings without target-side retuning. The authors ask when a playbook learned in one setting remains useful after transfer across target domains, model scales, model families, or runtime regimes.

The central finding: Prompt-side playbooks transfer only conditionally. The results across three benchmarks are "most consistent with an operational-compatibility view: transfer works when the assumptions encoded in a playbook remain compatible with the target deployment setting. Those assumptions include workflow, target capability, decoding regime, and runtime budget. When they remain aligned, transfer can improve success and sometimes reduce trajectory length. When they do not, the same playbook can inject the wrong action pattern, waste turns, or destabilize stopping behavior."

The paper's practical takeaway: teams should treat playbook reuse as a validation problem, not as a reuse-by-default strategy. Frozen transfer is framed "as a cold-start or amortized candidate when target-side failure trajectories are not yet available, when collecting target traces creates privacy, cost, or latency constraints, or when an existing artifact can be screened before investing in target-side construction."

The shared distill–validate–transfer protocol works as follows: the authors repeatedly run a source agent, mine all-fail and mixed-outcome bundles, and give them to a stronger closed-source distiller: Claude Opus 4.6 via Claude Code or GPT-5.4 via Codex, depending on the artifact family. The distiller abstracts recurrent rules, search priorities, and recovery heuristics into a compact playbook, validates it on the source side, and we then freeze it before transfer. The main controls are no-playbook baselines, same-length placebos, and matched-versus-mismatched transfer conditions.

Three benchmarks play complementary roles: ALFWorld is the controlled existence test for scene OOD, target scale, and cross-family transfer under matched decoding; TAU2-Bench is the main service-agent benchmark, spanning airline, retail, and telecom under both in-family and cross-family transfer; XBench-DeepSearch is a small runtime-shift stress test probing 32K-era playbooks under later 128K execution settings.

On ALFWorld, the answer is yes, but the 400-run audit sharpens that result: portability is also decoding-dependent.

Under greedy decoding (t=0): transferred playbooks help across scene-OOD evaluation, target scale, and model family, with gains growing with target capacity. Gains are modest on Qwen2.5-7B-Instruct, uniformly positive on Qwen2.5-32B-Instruct, and especially strong on Qwen3-32B and Llama3.1-70B-Inst, where the best lift reaches +23.6 pp. These gains come with shorter interaction traces rather than cost inflation.

Decoding robustness depends on capacity and artifact specificity: At t=0.3, On Qwen2.5-7B-Instruct, the Codex-7B-detailed artifact (X7D) flips from positive at t=0 to clearly harmful at t=0.3... hit-50 rises from 58.1% to 79.8% on seen, while mean trajectory length jumps from 32.8 to 42.4 steps. On Qwen2.5-32B-Instruct, X7D stays robust while the shorter X7C collapses. Qwen3-32B and Llama3.1-70B show the opposite pattern: sampling improves the baseline and keeps all four playbooks positive. Thus robustness depends on decoding regime × target capacity × artifact specificity.

Distilled guidance outperforms fixed demonstrations: Comparing X7D against a five-example fixed few-shot prompt (FS-5) on Qwen3-32B valid unseen under near-matched artifact budgets (4,191 vs. 4,001 bytes), X7D succeeds on 107/148 games (72.3%) versus 91/148 (61.5%) for FS-5: +10.8 pp, CI95 [+2.7,+18.9], exact McNemar p=.0139. The paper is careful to note this establishes incremental value over fixed demonstrations in one compatible transfer setting; it does not establish superiority over human-authored guidance or dynamic retrieval.

Illustrative ALFWorld cases show the dominant mechanism: task-relevant search priors are corrected first, and task-specific action templates are corrected second — e.g., rescuing episodes that would otherwise exhaust the 50-step ceiling.

TAU2-Bench is the main service-agent result because it matches deployment heterogeneity across business domains, target scales, and model families.

In-family landscape: The full Qwen2.5-Instruct grid covers 81 source–target–domain configurations. Cell-wise intervals are often wide, and only one in-family route survives Holm correction over this family. Point estimates are non-uniform, with visible positive regions in 32B/retail, 32B/airline, and 7B/telecom, but these regions are not individually confirmatory findings.

Matched-domain advantage: The grid provides a control against a pure prompt-injection explanation. Comparing the same injected file on its matched domain vs. two mismatched domains (controlling prompt length and format within source), Matched-domain transfer is stronger by +1.72 pp on average (CI95 [+0.25,+2.97], exact blocked-permutation p=.035; Appendix D.5). It supports a modest average domain-alignment advantage, not improvement on most routes.

Multiplicity correction: Global Holm correction over Pass1 retains 1/135 positive route; family-specific Holm additionally retains two cross-family negative routes, which also appear in a global Benjamini–Hochberg sensitivity analysis. The paper stresses: Failure to survive correction is not interpreted as evidence of zero effect. The aggregate contrast is the primary TAU2 inference, while the grid describes compatibility-sensitive heterogeneity.

Cross-family landscape: Point estimates do not fit a monotonic 'farther family, weaker effect' story; instead, signs vary by target and domain. After family-specific Holm correction, two negative routes remain, but neither survives global Holm.

Placebo control: A same-length generic placebo (structure-matched rewrite into generic workflow advice) stays near baseline while the transferred playbook reproduces the corresponding point-estimate direction, weakening a pure prompt-length account, suggesting imported procedural content matters.

Cost profile: When transfer helps, it is usually cost-neutral: message counts and agent-turn counts change only slightly, and most token overhead comes from fixed prompt injection. Negative configurations show performance drops while cost rises.

Frozen transfer vs. target-side redistillation: For a Qwen2.5-14B target, comparing no playbook, a target-derived same-domain artifact, and same-domain artifacts transferred from 7B or 32B sources, No strategy dominates across all three domains. The paper concludes: Frozen transfer is therefore a viable cold-start candidate when target-side failures are unavailable or costly to collect, not a preferred default.

This is the smallest benchmark and an exploratory stress test of one primary frozen artifact (distilled from Tongyi-DeepResearch-30B-A3B 32K source-side trajectories) across runtime conditions.

Accuracy is mixed: Under same-context transfer, Pass@1 rises by +4 to +8 pp and Pass@3 is flat or only slightly lower. Under cross-context transfer, Pass@1 is mixed (0 to +6 pp) but Pass@3 falls by 6 to 16 pp.

Operational behavior destabilizes: Under Tongyi-DR 32K, the playbook is roughly cost-neutral. Under Tongyi-DR 128K, Pass@1 is flat but Pass@3 drops by 16 pp. Under Qwen3-32B 128K, "the same artifact becomes much less bounded: search calls rise from 5.9 to 15.7, total tool calls from 10.4 to 24.7, and overlong-trajectory rate (>30T) from 8% to 25%."

Behavioral audit: A 150-pair audit localizes the shift: trajectories with an identical repeated query rise from 17.3% to 39.3%, those with at least three identical searches rise from 8% to 24%, and submissions within ten tool calls fall from 78% to 30%. Question-cluster intervals exclude zero, supporting requerying and delayed stopping as a proximal behavioral pathway, not an internal causal mechanism.

Redistillation is not automatically safer: Target-side redistillation... leaves Qwen3-32B 128K accuracy statistically unresolved while raising average total tool calls from 10.4 to 36.1 (+247%), with about 24% of trajectories reaching a budget.

Cost screening is inconclusive: In a prospective Gemini-2.5-Pro pilot with a frozen +5 mean-tool-call alert, smoke and held-out increases are +0.26 and +4.86; both narrowly agree on no alert, but the 347.1% held-out inflation demonstrates neither calibration nor a validated selector.

The paper's checklist for deployment:

• Validate transfer and redistillation on target-side slices, not source performance alone; prefer domain-matched candidates.

• Audit accuracy, termination, protocol compatibility, and cost, especially for strong targets or runtime changes.

• Freeze decision criteria and keep a rollback path; our pilot does not provide a universal threshold or selector.

• Prompt-side playbooks are deployment artifacts, not universally reusable knowledge modules.

The paper is explicit about scope: TAU2-Bench has limited per-cell power; the three-route placebo weakens a pure length explanation but is not a benchmark-wide causal intervention or a full prose/JSON/Markdown format ablation; the few-shot result does not establish cross-paradigm superiority; no comparison is made against human-authored playbooks or deployment-time dynamic retrieval systems; the XBench controlled runtime-shift result centers on one primary artifact–runtime pairing rather than independent artifact replications; and software engineering, code generation, and data analysis remain untested.

"Prompt-side playbooks transfer only conditionally. They help in controlled settings and beat fixed demonstrations in one comparison; TAU2 shows a modest matched-domain advantage but little corrected route-level evidence; and one XBench pairing exhibits re-querying, delayed stopping, and cost inflation. Redistillation is not automatically safer, but frozen transfer is not a preferred default. The contribution is diagnostic: both require target-side validation before rollout."

I can improve AI systems by building transfer-aware agent orchestration based on the paper's findings. The improved system can:

• Gate playbook reuse behind target-side validation. Before transferring a prompt-side playbook, the system runs a small target slice comparing three conditions: no playbook, same-length placebo, and transferred playbook. It activates the playbook only if transfer improves success without cost inflation; otherwise it rolls back. This prevents the paper's failure modes: injected wrong action patterns, wasted turns, and destabilized stopping.

• Check operational compatibility before deployment. The system evaluates whether the playbook's assumptions match the target on five axes: workflow, target capability, decoding regime, runtime budget, and model family. It rejects or adapts playbooks when the target uses a different decoder temperature, context-window regime, or tool-call budget. For example, it will degrade or strip a detailed playbook when target capacity is low and sampling temperature is high, avoiding the Qwen2.5-7B failure where t=0.3 flipped a positive artifact harmful.

• Prefer domain-matched playbooks and rank candidates by domain alignment. The system scores candidate playbooks by source-domain similarity and uses matched-domain artifacts first, because matched transfer is stronger by about +1.7 pp on average. It avoids injecting a playbook from an unrelated domain when a matched one exists, and it treats mismatched routes as unvalidated rather than automatically beneficial.

• Monitor operational behavior continuously, not just accuracy. During execution, the system tracks trajectory length, repeated queries, identical search calls, tool-call counts, and submission timing. If behavioral drift exceeds thresholds—such as search calls climbing from 6 to 16, or overlong-trajectory rate rising from 8% to 25%—the system terminates the playbook, falls back to baseline, and flags the artifact as incompatible with the current runtime. This catches the XBench failure mode where a frozen playbook caused re-querying, delayed stopping, and cost inflation.

• Choose distilled procedural guidance over fixed demonstrations when budgets match. The system can generate or select condensed, rule-based playbooks rather than large fixed few-shot prompts, since distilled guidance improved success from 61.5% to 72.3% at comparable byte budgets in a compatible transfer setting. It will reserve fixed demonstrations only when distillation is unavailable or validation shows them competitive.

• Treat redistillation as a candidate, not a default. When target-side failures are available and validation shows frozen transfer failing, the system can attempt target-side redistillation—but it will first estimate cost and accuracy risk, because redistillation may raise tool calls by 247% without resolving accuracy. It compares redistilled, frozen-transferred, and baseline policies on target slices and selects the cheapest one that meets the success threshold.

• Report uncertainty and corrected inference. The system will not promote a transferred playbook based on a single positive cell. It computes confidence intervals, multiplicity-corrected p-values, and blocked permutation tests across source–target–domain configurations. It marks a playbook as confirmed only if it survives correction or if the aggregate matched-versus-mismatched contrast is positive; otherwise it labels the playbook as promising but unconfirmed and requires further validation.

• Detect cold-start conditions and decide reuse policy. The system tracks whether target-side failure trajectories are available, whether collection is costly, and whether privacy or latency constraints forbid it. In genuine cold-start settings, it may use frozen transfer as an amortized candidate; once target traces accumulate, it automatically revalidates and can switch to target-derived artifacts or redistillation.

• Maintain a rollback path and freeze decision criteria. Every playbook deployment is wrapped in a reversible policy: the system records the decision criteria, keeps the baseline agent intact, and can disable the playbook mid-run if behavioral monitors trigger. It does not treat prompt-side playbooks as universal knowledge modules, but as deployment artifacts with expiry conditions tied to runtime and model configuration.

The improved system can therefore reuse prompt-side playbooks safely, avoid harmful transfer, keep costs bounded, and know when to say this artifact does not transfer here.

Prompt-side playbooks can improve tool-using language agents without retraining, but their portability beyond the source setting is unclear. We study frozen playbook transfer under a shared distill--validate--transfer protocol. On ALFWorld, transfer is beneficial under controlled greedy decoding and, in one near-budget-matched comparison, distilled guidance outperforms five fixed demonstrations. On TAU2-Bench, a prespecified aggregate contrast supports a modest average matched-domain advantage, but global Holm correction retains only one of 135 route-level effects; the remaining grid provides descriptive evidence of compatibility-sensitive heterogeneity. On XBench-DeepSearch, one artifact--runtime pairing preserves useful first-try heuristics while producing repeated queries, delayed stopping, and substantial cost inflation after a context-runtime shift. Across benchmarks, transferred and target-derived playbooks both require target-side validation of success, termination, protocol compatibility, and cost. Frozen transfer is therefore a conditional cold-start option, not a reuse-by-default strategy or a universally preferable alternative to target-side redistillation.