PromptOps: Iterative Prompt Optimization for LLM-as-a-Judge Systems
cs.AI, cs.LG
Submitted: 2026-11-04
Updated: 2026-11-04
Terminology
Sources
- When Generic Prompt Improvements Hurt: Evaluation-Driven Iteration for LLM Applications
- SedarEval: Automated Evaluation using Self-Adaptive Rubrics
- EvoPrompt: Connecting LLMs with Evolutionary Algorithms Yields Powerful Prompt Optimizers
- Beyond One-Size-Fits-All: Inversion Learning for Highly Effective NLG Evaluation Prompts
- DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines
- Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting
- HPSS: Heuristic Prompting Strategy Search for LLM Evaluators
- LLM Prompt Duel Optimizer: Efficient Label-Free Prompt Optimization
- RevisEval: Improving LLM-as-a-Judge via Response-Adapted References
Related papers
- Multi-Objective Structured Pruning of LLMs for Latency and Model Size Optimization
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Embeddings for Preferences, Not Semantics
- Can We Trust Item Response Theory for AI Evaluation?
- Diagnosing Task Insensitivity in Language Agents