Learning, Fast and Slow: Towards LLMs That Adapt Continually

The gist Large language models (LLMs) can be adapted continually by introducing a fast-slow learning framework that separates task-specific information into fast weights and persistent general behaviors into slow weights This framework, Fast-Slow Training (FST), is shown to be more sample efficient, preserves plasticity, and enables continual learning in LLMs compared to parameter-only training methods.

The Fast-Slow Learning Framework

The paper introduces a fast-slow learning framework for LLMs where model parameters are treated as “slow” weights and optimized context is treated as “fast” weights The slow parametric component, denoted by θ, represents the model weights which are expensive to update, persist across tasks, and encode long-lived behavior. The fast textual component, denoted by ϕ, represents prompts and context which can be changed cheaply and frequently to capture task-level adaptation without permanently modifying the model. This division of labor allows the fast weights to learn from textual feedback to absorb task-specific information while allowing slow weights to stay closer to the base model and persist general reasoning behaviors.

Training Mechanism: Fast-Slow Training (FST)

Fast-Slow Training (FST) jointly optimizes slow weights θ through reinforcement learning and fast weights ϕ through reflective prompt evolution The method maintains a population of K textual prompts, Φ = ϕ(1),..., ϕ(K), and optimizes the joint objective J(θ, ϕ) = Ex∼D, ϕ∼U(Φ), y∼πθ(·x,ϕ)[r(x, y)]. Training proceeds in cycles of T slow-weight updates where GEPA optimizes the fast weights Φc+1 using the current policy πθc as the rollout model. For the next T steps, θ is updated on minibatches from Lc while holding Φc+1 fixed. During this slow-weight update phase, rollouts are computed by sampling each prompt ϕ(k) ∈ Φc+1 exactly G/K times for each problem p, and rewards are normalized by per-problem statistics.

Key Advantages of FST

The fast-slow distinction yields several consequences across reasoning tasks

1. Fast textual adaptation improves data efficiency, as fast weights incorporate task-level signal rapidly, matching RL reward with up to 3× fewer rollouts and consistently reaching a higher performance ceiling.

2. Fast-slow training induces smaller slow-weight displacement, resulting in up to 70% lower KL divergence to the base policy at matched reward compared to RL-only baselines.

3. Fast-slow training preserves plasticity, as FST trained models adapt more effectively to subsequent tasks than parameter-only trained models, preventing catastrophic forgetting.

4. Fast-slow training enables continual learning where tasks change on the fly, allowing the model to adapt more quickly to changing objectives than parameter-only RL stalls.

Empirical Results and Mechanisms

The framework's benefits stem from several observations regarding the interaction between channels

Observation 1:

The fast channel does not have this latency: GEPA can extract task structure from a handful of rollouts and inject it through Φ immediately. This is because slow weights are slow in how many updates they require to begin moving signal at all, while the fast channel provides immediate gradient signal.

Observation 2:

FST does not assume a fixed division of labor between the two channels, and lets each task draw on whichever channel pays off while still combining them when both contribute. The in-distribution gain is decomposed into slow- and fast-weight contributions, showing that the joint cell (FST weights + FST prompt) dominates on HoVer-hard and CodeIO, while the slow weights carry most of the gain on Math (Polaris).

Observation 3:

FST escapes the zero-reward regime by step ∼50, an order of magnitude before RL begins to move signal at ∼250. This demonstrates that the fast channel provides enough gradient signal for FST to climb rewards quickly.

Conclusion

The paper concludes that effective post-training should be viewed as optimization over multiple adaptation channels, where fast textual weights and slow parametric weights are trained together to achieve rapid and task-specific improvements while preserving the generality and plasticity of the base model. FST offers a path toward post-training methods that are more data-efficient, less destructive, and more amenable to continual learning.

Limitations and Future Work

The authors note that while the study focuses on CISPO and GEPA as optimizers, studying the impact of changing the prompt or weight optimizer is an interesting avenue for future work. There is also potential to make the method more compute efficient and better reuse trajectories across prompt and weight optimization. Finally, a more comprehensive study of applying this paradigm to distillation-based approaches is considered an exciting avenue for future work.

Related Work

The paper situates FST within related fields by noting that the fast/slow decomposition predates deep learning, with roots in the neuroscience of complementary learning systems. It also discusses modern fast–slow methods for LLM RL, such as BetterTogether and LANPO, which interleave language and numerical feedback via per-instance reflections. The framework is related to other LLM-as-optimizer methods that use natural-language reflection or self-feedback.

Algorithm Details

The pseudocode for FST shows the interleaved nature of the optimization, where GEPA returns the top-K candidates from its Pareto frontier to seed the next RL phase. The training involves cycling through T slow-weight updates, with each cycle involving a GEPA phase to generate fast weights Φc+1. The rollout group for the RL update is formed by sampling from the current population of prompts Φc+1, mixing prompt and sampling variation within the same advantage computation.

Star-Graph Dataset Construction

The paper describes a synthetic dataset construction method called Star-graph search, which is used for planning tasks. Each instance is parameterized by a triple (d, p, n): source degree d, path length p, and node-pool size n. This construction creates hard exploration problems where the first hop is the only real branching decision.

Hyperparameters and Compute

The appendix lists detailed hyperparameters for reproducing results, including settings for GRPO, cispo surrogate loss, and GEPA configurations. The headline runs in the main paper utilize single-node configurations with 8× H100 GPUs. The computation time for a single RL+FST run is on the order of 25–40 H100-GPU-hours, with GEPA cycles adding a sizable fraction to this cost.

Final Summary

Fast-Slow Training (FST) is a fast and slow learning framework for LLMs that jointly optimizes slow model parameters via RL and fast textual context via reflective prompt evolution. This approach reaches matched performance with 1.4–3× fewer optimizer steps, attains a higher asymptote, and incurs lower KL displacement than RL alone, which in turn translates into preserved plasticity and stronger continual-learning behavior on new tasks. The results suggest that effective post-training should not ask model parameters to absorb all forms of adaptation <ref:2605.

1. Bold Header: Fast-Slow Training (FST) for Data Efficiency

FST is up to 3× more sample-efficient than only slow learning (RL) across reasoning tasks while consistently reaching a higher performance asymptote. This allows LLMs to reach peak performance with substantially fewer samples by leveraging the fast textual channel, which incorporates task-level signal rapidly.

2. Bold Header: Increased Performance Asymptote

FST induces a higher ceiling for model performance compared to parameter-only methods, as evidenced by FST’s fitted asymptote exceeds RL’s across all tested tasks on CodeIO, Math (Polaris), and HoVer-hard. This suggests that Pushing part of the task adaptation into the textual fast-weight channel Φ in addition to the slow weights θ helps the overall method converge to a higher accuracy ceiling.

3. Bold Header: Reduced Catastrophic Forgetting

FST leads to smaller slow-weight displacement, resulting in models that remain closer to the base LLM (up to 70% less KL divergence). This reduced drift preserves plasticity and ensures that after training on one task, FST trained models adapt more effectively to a subsequent task than parameter-only trained models.

4. Bold Header: Enhanced Continual Learning Capability

FST enables continual learning by allowing the model to continue to acquire each new task while parameter-only RL stalls. This is demonstrated in multi-task training where FST reaches near-peak in every stage, mirroring the data efficiency gap of Section 4 Advantage 1.

5. Bold Header: Data Efficiency via Contextual Learning

The framework allows for data efficient learning because the fast weights incorporate task-level signal rapidly, enabling systems to improve without waiting for slow parameter updates. This overcomes the widely accepted 1-bit-per-episode information limit of binary RLVR.

6. Bold Header: Joint Optimization of Adaptation Channels

FST jointly optimizes slow weights through RL and fast weights through GEPA, allowing them to co-evolve. This is formalized by the objective function maximizing J(θ, ϕ) = Ex∼D, ϕ∼U(Φ), y∼πθ(·x,ϕ)[r(x, y)], meaning adaptation is distributed appropriately across both elements.

7. Bold Header: Adaptive Prompt Engineering

The fast weights are optimized via GEPA, which uses a reflective evolutionary procedure to maintain a Pareto frontier of complementary prompts. This allows the system to use different prompts perform best on different subsets of D, providing the policy access to multiple conditioning behaviors simultaneously.

Large language models (LLMs) are trained for downstream tasks by updating their parameters (e.g., via RL). However, updating model parameters forces them to absorb task-specific information, which can result in catastrophic forgetting and loss of plasticity. In contrast, in-context learning with fixed LLM parameters can cheaply and rapidly adapt to task-specific requirements (e.g., prompt optimization), but cannot by itself typically match the performance gains available through updating LLM parameters. Inspired by how human learning works at multiple time scales (e.g., System 1 vs 2), we propose a fast-slow learning framework for LLMs, instantiated with model parameters as "slow" weights and optimized context as "fast" weights. These fast "weights" can learn from textual feedback to absorb the task-specific information, while allowing slow weights to stay closer to the base model and persist general reasoning behaviors. Fast-Slow Training (FST) is up to 3x more sample-efficient than only slow learning (RL) across reasoning tasks, while consistently reaching a higher performance asymptote. Moreover, FST-trained models remain closer to the base LLM (up to 70% less KL divergence), resulting in less catastrophic forgetting than RL-training. This reduced drift also preserves plasticity: after training on one task, FST trained models adapt more effectively to a subsequent task. In continual learning scenarios, where task domains change on the fly, FST continues to acquire each new task while RL stalls.