Prompt Stability Matters: Evaluating and Optimizing Auto-Generated Prompt in General-Purpose Systems

The gist: This work introduces prompt stability as a key factor for building robust and effective prompt generation systems in general-purpose multi-agent systems, arguing that consistency across repeated executions is more crucial than immediate task performance

Motivation and Problem Statement

Automatic prompt generation plays a crucial role in enabling general-purpose multi-agent systems to perform diverse tasks autonomously. Existing methods typically evaluate prompts based on their immediate task performance, overlooking the intrinsic qualities that determine their reliability. This outcome-centric view not only limits interpretability but also fails to account for the inherent stochasticity of large language models (LLMs). In multi-agent systems where agents depend on one another’s outputs, such inconsistencies can cascade. Thus, we argue that prompts should be evaluated not only for correctness or accuracy, but also for stability—their ability to consistently elicit semantically coherent responses across executions.

Theoretical Analysis of Prompt Stability

The analysis considers a planner–executor structure where the deviation between the envisioned and actual outputs is driven by the variance of executor responses. The deviation between actual and intended system outputs is given by s − ˆs = u T (X − X̂)v, which simplifies to Xn i=1 uivi(xi − xˆi). Lemma 1 establishes a bound on the probability that this deviation exceeds a threshold epsilon, showing s − ˆs P (s − ˆs ≥ ϵ) ≤ 2 exp-epsilon squared / (2 Pn i=1(uivi) squared Var(xi)). This analysis shows that the deviation between actual and intended system outputs is primarily driven by the variance of executor responses.

Semantic Stability as a Metric

To capture the consistency of outputs generated from the same prompt, semantic stability S(p) is defined as the average pairwise cosine similarity of sampled outputs. This metric is computed using embeddings, where each sampled output yi is encoded into a semantic vector vi = ϕ(yi). The definition of semantic stability S(p) is S(p) = 1 − 2 / N(N − 1) X i<j d ij, where d ij is the cosine distance between output pairs. Higher values of S(p) indicate greater semantic consistency and thus stronger prompt stability. Empirical results show a strong positive correlation (r = 0.73) between semantic stability and task success rate, confirming that more stable prompts are significantly more likely to succeed.

Stability-Guided Optimization Framework

The system jointly optimizes two objectives: (1) Stability Objective: max pi S(pi), which reduces execution variance via stability-aware prompt generation; and (2) Planner Alignment Objective: min pi s⋆′− ˆs, which improves task decomposition by minimizing the gap between planner intent and idealized output. To reduce execution deviation, prompts with S(pi) < τ trigger a Reviewer Agent that diagnoses instability and revises the unstable subcomponent z ∈ q i, i i, k i, or h i to form an updated prompt p'i. To reduce planner error s⋆′− ˆs, mechanisms like Subtask Optimization adjust subtask boundaries and the Plan Updater injects corrective signals ∆t into future subtasks based on execution feedback.

Empirical Validation

The framework was tested using GPT-4o as the language model backbone across general-purpose tasks like math reasoning, data analysis, code generation, and machine learning. Promptor outperformed baselines such as AutoGen and DI in mathematical reasoning (MR) and achieved a high similarity score of 0.88 on the InfiAgent-DABench benchmark for data analysis (DA). In professional domains, Promptor achieved an accuracy of 86% on single-cell RNA sequencing (scRNA-seq) analysis and the highest accuracy among tested systems on the SMILES-to-molecular formula (S2MF) task in chemistry. Ablation studies confirmed that each module in the framework plays a critical role, and replacing semantic stability with token-level KL divergence hurt results, confirming S(p) as a more effective refinement signal.

Conclusion

This paper introduces prompt stability as a core design principle for general-purpose multi-agent systems. Through theoretical analysis and empirical validation, we show that semantic stability provides a practical and interpretable proxy for measuring output variance and guiding prompt optimization. Our proposed system, Promptor, integrates stability-aware refinement and feedback-driven planning to improve execution consistency and task success. These findings suggest that prompt stability offers a robust foundation for scaling multi-agent coordination in LLM-based systems.

--- Page 1 ---

arXiv:2505.13546v1 [cs.AI] 19 May 2025

Prompt Stability Matters: Evaluating and Optimizing Auto-Generated Prompt in General-Purpose Systems

Ke Chen Yufei Zhou Xitong Zhang Haohan Wang

Abstract

Automatic prompt generation plays a crucial role in enabling general-purpose multi-agent systems to perform diverse tasks autonomously. Existing methods typically evaluate prompts based on their immediate task performance, overlooking the intrinsic qualities that determine their reliability. This outcome-centric view not only limits interpretability but also fails to account for the inherent stochasticity of large language models (LLMs). In this work, we bring attention to prompt stability—the consistency of model responses across repeated executions—as a key factor for building robust and effective prompt generation systems. To quantify this, we propose semantic stability as a criterion for assessing the response consistency of prompts, and fine-tune a LLaMA-based evaluator to measure it automatically across tasks. These components have enabled us to develop the first stability-aware general-purpose prompt generation system that leverages stability feedback to iteratively enhance both prompt quality and system-level performance. Furthermore, we establish a logical chain between prompt stability and task success by analyzing the structural dependencies within our system, proving stability as a necessary condition for effective system-level execution. Empirical results across general and domain-specific tasks demonstrate that our stability-aware framework improves both accuracy and output consistency. By shifting the focus from one-off results to persistent reliability, our work offers a new perspective on prompt design and contributes practical tools for building more trustworthy general-purpose systems.

1 Introduction

Imagine a project manager acting as a planner. The project’s success hinges on the clarity of these instructions—any ambiguity may lead to misinterpretation, misalignment, and ultimately, failure to realize the original vision. Now, imagine the same scenario with a twist: both the project manager and the team are highly versatile, capable of handling tasks across diverse domains. This general-purpose setting magnifies the challenge, as domain-specific language increases the risk of misunderstanding. In such context, instructions must be precise, consistent, and unambiguous.

This is no longer hypothetical. While human teams rarely have such general-purpose abilities, AI agents increasingly do. General-purpose multi-agent systems built on large language models (LLMs) assign roles and responsibilities via prompts [5, 6, 8, 7], which define agent behavior, interaction and decision-making.

1. Prompt refinement via semantic stability: Implement a mechanism where prompts are iteratively revised until their semantic stability S(p) ≥ τ is met, as this serves as a practical proxy for output variance in stochastic LLM settings. This directly reduces execution deviation by targeting the variance of executor responses.

2. Joint optimization of stability and alignment: Design Promptor to simultaneously optimize two objectives: (1) Stability Objective: max pi S(pi) and (2) Planner Alignment Objective: min pi s⋆′− ˆs. This ensures that prompt quality is judged not only by outcome success, but also by the ability to achieve it consistently under varying conditions.

3. Dynamic subtask refinement: Implement a Plan Updater that injects a corrective signal ∆t into future subtasks based on execution feedback, refining the planner’s trajectory ˆs. This mechanism aims to minimize the gap between planner intent and the ideal output by adjusting subtask boundaries, types, and I/O specifications.

4. Structured output constraint: Enforce explicit input-output specifications for each subtask to constrain distributions such that xi ∼ Λ(i) (pi xi ∈ C). This restriction is designed to reduce semantic variance and improve stability across executions by limiting the range of possible outputs.

5. Context retention via summarization: Integrate a Summarizer Agent that distills raw outputs into structured summaries, which serve as compact memory units, enhancing context retention and reducing semantic drift in downstream subtasks. This ensures long-horizon tasks maintain coherence by preserving relevant information for subsequent steps.

Automatic prompt generation plays a crucial role in enabling general-purpose multi-agent systems to perform diverse tasks autonomously. Existing methods typically evaluate prompts based on their immediate task performance, overlooking the intrinsic qualities that determine their reliability. This outcome-centric view not only limits interpretability but also fails to account for the inherent stochasticity of large language models (LLMs). In this work, we bring attention to prompt stability-the consistency of model responses across repeated executions-as a key factor for building robust and effective prompt generation systems. To quantify this, we propose semantic stability as a criterion for assessing the response consistency of prompts. Based on the proposed metric, we developed the first stability-aware general-purpose prompt generation system that leverages stability feedback to iteratively enhance both prompt quality and system-level performance. Furthermore, we establish a logical chain between prompt stability and task success by analyzing the structural dependencies within our system, proving stability as a necessary condition for effective system-level execution. Empirical results across general and domain-specific tasks demonstrate that our stability-aware framework improves both accuracy and output consistency. By shifting the focus from one-off results to persistent reliability, our work offers a new perspective on prompt design and contributes practical tools for building more trustworthy general-purpose systems. The UI and source code for our system are publicly available at: https://xucheng63.github.io/MyDataPilot/.