PPDL: LLM-Based Flows as Probabilistic Programs
Louis Mandel, Guillaume Baudart, Mandana Vaziri, Martin Hirzel
cs.LG, cs.PL
Submitted: 2026-08-05
Comments: Published at ICML 2026
Code: https://github.com/IBM/prompt-declaration-language
License: http://creativecommons.org/licenses/by/4.0/
The gist: Building reliable applications that leverage large language models (LLMs) remains a significant challenge.
Terminology
Abstract
Building reliable applications that leverage large language models (LLMs) remains a significant challenge. While LLMs offer impressive capabilities across diverse tasks, their outputs often lack accuracy and provide no clear measure of confidence. This uncertainty compounds in flows of multiple calls to LLMs and other tools, making it difficult for developers and end-users to trust the results. This paper introduces a probabilistic language for programming LLM-based flows. It enables developers to quantify and propagate uncertainty throughout the application's flow, and experiment with different inference scaling techniques without adding a single line of code beyond the flow's logic. We present an experimental study to demonstrate this capability, and a case study building a theorem proving agent for the Rocq theorem prover.
Sources
- GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning
- Program Synthesis with Large Language Models
- Training Verifiers to Solve Math Word Problems
- Large Language Monkeys: Scaling Inference Compute with Repeated Sampling
- Magnushammer: A Transformer-Based Approach to Premise Selection
- Generative Language Modeling for Automated Theorem Proving
- FEVER: a large-scale dataset for Fact Extraction and VERification
- MiniF2F: a cross-system benchmark for formal Olympiad-level mathematics
- Hilbert: Recursively Building Formal Proofs with Informal Reasoning
- PDL: A Declarative Prompt Programming Language
- MiniF2F in Rocq: Automatic Translation Between Proof Assistants -- A Case Study
- Proving Theorems Recursively
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks