ProDVI: Programmatic Dynamics Priors for Value Network Initialization

The paper proposes ProDVI (Programmatic Dynamics Priors for Value Network Initialization), a framework that leverages the commonsense and domain knowledge encoded in large language models (LLMs) to initialize reinforcement learning (RL) agents, without relying on pre-collected datasets, high-fidelity simulators, or curated task distributions. The paper addresses the longstanding challenge that RL agents are typically initialized from scratch, forcing them to acquire task-relevant knowledge through online interaction, which contributes to deep RL's notorious sample inefficiency.

The authors note that existing approaches to informative initialization — offline-to-online RL, sim-to-real transfer, and meta-RL — require pre-collected datasets, high-fidelity simulators, or a curated distribution of related tasks, which impose non-trivial barriers. This motivates the central question: Can we obtain informative initializations for RL agents without relying on these non-trivial prerequisites? The paper answers this by exploring whether knowledge encoded in LLMs can be used to provide informative parameter initializations for RL agents.

ProDVI addresses three design questions:

1. What kind of prior knowledge should be elicited from LLMs? ProDVI prompts a code-generating LLM to generate prior knowledge about the dynamics of the environment in the form of executable Python functions. Given randomly sampled state-action pairs, the generated program (denoted gLLM) produces approximate next-state predictions. The environment is described in a code-based format: a Python class whose docstrings describe the action and observation spaces (including semantic meaning of each dimension) and whose internal variables store environment configuration parameters (e.g., gravity, body masses) when available. The LLM is prompted in stages: first to generate a textual dynamics analysis decomposing how the current observation is affected by the action and how the current observation evolves into the next observation step by step, then to implement this analysis as executable Python code implementing the do simulation function.

2. Where should the prior be injected? The prior is injected into the state-action encoder of the value network within an actor-critic RL framework. This connects ProDVI to prior dynamics-based representation learning methods such as OFENet, which trains state-action representations by predicting the next observation.

3. How should it be distilled? ProDVI first generates synthetic transitions. Observations are sampled from a multivariate Gaussian and clipped (o ∼ N(0, I), õ = clip(o, −O, O)), actions are uniformly sampled from their bounds (a ∼ U(amin, amax)), and the pseudo next observations are computed as ô(t+1) = gLLM(õt, at). These are normalized and used to pretrain the state-action encoder with an auxiliary dynamics prediction loss: LAux = ∥dϕ(zt) − õ(t+1)∥22. Importantly, the goal is not to generate samples from the true observation-action distribution but rather to construct a diverse set of input-output pairs generated by gLLM. Following prior work, pretraining and online RL are performed in a normalized observation space.

A soft reset is applied to the pretrained parameters to mitigate plasticity loss, following the shrink-and-perturb strategy: θinit = αθ̄ + (1 − α)θ0, where θ0 is freshly sampled from the original initialization distribution and α ∈ (0, 1) controls retention of pretrained parameters.

1. No expensive prerequisites: "It does not require pre-collected trajectories, high-fidelity simulators, or meta-training tasks. The only external resources required by ProDVI are access to a code-generating LLM and metadata describing the target environment."

2. No faithful simulation required: "Because the generated dynamics functions are used only for dynamics-aware representation pretraining, they do not need to serve as faithful simulators of the target environment... even imperfect programs can provide informative structural biases."

3. LLM decoupled from online RL: Once the dynamics programs have been generated, the LLM is no longer queried, avoiding repeated LLM calls.

4. Reuse across tasks with shared dynamics: Because ProDVI focuses on dynamics priors rather than any specific task, the generated priors are specific to the underlying dynamics while remaining agnostic to individual tasks.

• Environments: 5 locomotion tasks from OpenAI Gym (Ant-v5, HalfCheetah-v5, Hopper-v5, Humanoid-v5, Walker2d-v5) and 7 tasks from DMControl dog and humanoid domains. Gym agents trained for 1M time steps; DMControl agents for 500k steps (1M frames, action repeat 2).

• LLM: GPT-5.5 is used as the code-generating LLM.

• Backbone: ProDVI adopts AnonMethod (a recent model-free method built on TD3 with observation-predictive representation learning and NormMethod normalization) as the backbone; also tested with a SAC-based variant.

• Hyperparameters: 7M synthetic transitions and α = 0.5 as defaults, selected via a sensitivity study.

• Evaluation: 5 seeds per experiment; agents evaluated every 5k environment steps over 10 episodes. Gym scores are Deep-TD3-normalized (or Deep-SAC-normalized); DMControl scores are raw episode returns (max 1000). Mean and IQM are reported as aggregate metrics.

Improvement over strong baselines: ProDVI improves both AnonMethod and AnonMethod w/o Aux on Gym and DMControl. Gains are especially pronounced in the low-data regime, for example at 0.25M environment steps, which indicates that ProDVI can effectively improve sample efficiency. For instance, on Gym at 0.25M steps, AnonMethod w/o Aux improves in Mean from 0.87 to 1.20 and IQM from 0.79 to 1.11 with ProDVI; on DMControl at 0.25M, Mean improves from 359 to 396 and IQM from 308 to 351. The authors note these gains suggest that ProDVI is not merely compensating for the absence of representation learning. Instead, it provides an informative initialization before real environment interaction.

Transfer to SAC-based algorithms: ProDVI improves both AnonMethod-SAC w/o Aux and AnonMethod-SAC variants, suggesting that the dynamics priors learned from LLM-generated programs are not specialized to TD3-based methods.

Why the priors help: The paper rules out the accuracy-based explanation. Table 3 shows LLM-generated programs have high MSE and MAPE prediction errors on Gym tasks, leading to the conclusion that the dynamics encoded by these programs are far from the true environment dynamics. This finding rules out the accuracy-based explanation. Instead, the paper shows via a dynamics prediction experiment (Table 4) that "the initialization obtained after dynamics priors distillation leads to lower training losses than training from scratch in most tasks and update budgets. This suggests that although the LLM-generated dynamics priors are not accurate dynamics models, they still contain informative information for dynamics modeling."

Robustness to LLM stochasticity: Two additional independently generated dynamics-prior programs (Prior #2 and Prior #3) both improve aggregate performance on Gym and DMControl, demonstrating that ProDVI is robust to the stochasticity of LLM generation and remains effective when using dynamics priors generated from different runs.

The paper concludes that ProDVI improves the sample efficiency and performance of strong model-free RL algorithms and that programmatic dynamics priors provide a promising way to use LLM knowledge for sample-efficient RL. The stated limitations are: (1) ProDVI is currently designed for state-based settings. Extending ProDVI to more complex settings, such as high-dimensional visual observations, remains an important direction for future work; and (2) "the quality of the generated dynamics priors is still constrained by the capabilities of the LLM. These programs are not guaranteed to capture informative task-relevant dynamics regularities. This limitation is expected to be mitigated as LLMs continue to improve in code generation, physical reasoning, and domain-specific understanding."

The paper distinguishes ProDVI from LLM-generated world models (e.g., Code World Models, WorldCoder): "ProDVI does not aim to construct a faithful world model for planning. Instead, it uses the generated dynamics programs before online RL begins, producing synthetic transitions to pretrain the state-action encoder of the value network." It also positions ProDVI as orthogonal to other LLM-enhanced RL methods (Eureka for reward design, LORO for off-policy data warm-starting, LESR for state engineering, LaRe for latent rewards, LLM-Explorer for exploration guidance) since ProDVI targets network parameters rather than rewards, initial experience, input states, or exploration strategies.

Based on ProDVI, here are the specific improvements to AI systems and what the improved system can do:

• Sample-efficient RL from scratch: The AI system can acquire task-relevant value network representations before any real environment interaction, using only an LLM-generated dynamics program. This leads to faster convergence and higher returns, especially in low-data regimes (e.g., 38% higher mean score on Gym at 0.25M steps).

• No reliance on expensive prerequisites: The system does not need pre-collected offline datasets, high-fidelity simulators, or curated meta-training task distributions. It only requires a code-generating LLM and metadata describing the environment (e.g., observation/action dimensions and semantics).

• Robust to imperfect world models: Because the LLM-generated dynamics program is used only to pretrain a dynamics-aware encoder (not as a faithful simulator), the system can benefit from even highly inaccurate programs. This means it can work even when the LLM's physical reasoning is imperfect.

• Seamless integration with existing actor-critic methods: The improved system can be plugged into strong model-free backbones (e.g., TD3-based AnonMethod or SAC-based variants) by injecting a distilled dynamics prior into the state-action encoder and adding an auxiliary next-state prediction loss. It does not require changing the online RL algorithm.

• LLM-decoupled online inference: The LLM is queried only once to generate the dynamics program. After that, online RL proceeds without further LLM calls, making the system practical and cost-efficient.

• Transferable across tasks with shared dynamics: The prior is task-agnostic but dynamics-specific, so the same pretrained initialization can be reused for different reward functions within the same environment (e.g., different locomotion objectives), enabling rapid adaptation.

• Mitigation of plasticity loss: By applying a soft reset (shrink-and-perturb) to pretrained parameters, the system retains useful structure while preserving the ability to learn new information, avoiding the common pitfall of pretrained representations that become rigid during online fine-tuning.

• Improved representation learning quality: The system's dynamics-pretrained encoder yields lower training losses when fine-tuned online, indicating that the initialization provides a better starting point for modeling environment dynamics than random initialization — even when the LLM prior itself is not an accurate dynamics model.

In summary, the improved AI system can achieve sample-efficient deep RL in state-based environments with minimal prerequisites, leveraging imperfect LLM-generated dynamics priors to initialize value networks effectively, and can transfer these priors across related tasks while remaining compatible with mainstream actor-critic algorithms.