Continual Learning without Continual Training

arXiv:2610.10379 · cs.LG · Submitted 2026-10-07 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "Continual Learning without Continual Training".

Jane: The Latent Concept PFN model proposes replacing continual training with continual inference by adapting to new domains and classes through in-context evidence sets rather than parameter updates.

Tom: First, who's behind it and why it matters.

Title and authors: Tom: Let's recap what we just discussed about "Continual Learning without Continual Training." Essentially, we’ve established that this method works by using in-context evidence sets instead of parameter updates.

Jane: Right. It avoids the massive cost of retraining every time we get new data or a different task because you aren't touching any weights during the stream phase.

Jane: That points toward how we can keep AI systems useful in real-world environments by avoiding that constant retraining overhead.

The paper's summary: Tom: Now let's look at what the authors summarize in "Continual Learning without Continual Training." They basically say this is about replacing constant retraining with just feeding new context through a frozen model.

Jane: Right, it’s about replacing constant retraining with just feeding new context through a frozen model. It avoids that massive cost every single time we get new data or a different task.

Tom: The main takeaway is you can get competitive accuracy against full-data training methods without touching a single weight during inference.

Jane: It also gives us that concept read-out tool, which shows exactly where the model starts losing its footing when it hits something new.

Tom: Exactly, the separation between that frozen model and the growing memory lets it handle new classes and domains quite well.

Jane: That changes how we think about keeping AI useful in real-world environments because we don't have to plan a full retraining cycle for every single update.

Tom: The Latent Concept PFN really shows that structure learned early on can be reused compositionally when new data comes along.

Jane: It’s a solid piece of work showing how representation learning can be more flexible than just brute-force training updates.

Tom: So we've covered this paper on "Continual Learning without Continual Training."

Jane: Right, it’s about replacing constant retraining with just feeding new context through a frozen model.

The paper's improvements: Tom: Now let's look at what the authors suggest to improve this system further and what those changes mean for us on the ground. They point out that having concept annotations required at meta-training time is a limitation, so future work needs to explore ways around that using tools like vision-language models or post-hoc analysis instead of relying on them upfront.

Jane: That’s a major point because if you need pre-labeled data just to set up the initial latent space, it adds a lot of friction before you even start learning continuously.

Jane: They also pointed out that while the concept read-out is great for identifying where errors happen, it can’t actually correct those concepts in real time like a conventional continual learning method could.

Tom: So the improvement isn't about fixing the model's internal concepts immediately, it's more about better diagnostics for where the failures are coming from during inference.

Jane: Exactly. And they suggest that even though concept annotations are needed initially, there are methods out there using vision-language models or post-hoc analysis to bypass that requirement entirely in the future.

Tom: From an engineering standpoint, if you can get around needing those upfront concept labels, it simplifies the entire pipeline significantly because you remove that dependency on perfectly annotated training sets for every new task or domain.

Jane: And they also mentioned that the model's performance is best when it’s dealing with a prior shift scenario, like in medical imaging data such as CXR, and in class-incremental settings where those concept annotations are most helpful.

Conclusion: Tom: So we've covered this paper on "Continual Learning without Continual Training," and we’ve established that this system works by using in-context inference to adapt without updating model parameters during the continual learning stream.

Lu: I think the focus on concept anchoring during meta-training is important because it sets up that stable latent space correctly for the subsequent stream adaptation.

Meng: For deployment, I’m still concerned about that memory growth—how much context do we need to keep growing before it becomes too big or slow?

Lu: From my side, the focus on concept anchoring during meta-training is important because it sets up that stable latent space correctly for the subsequent stream adaptation.

Meng: I just hope they can show us a way to manage that memory efficiently when we push this into a production environment that handles continuous data streams.

Lalam: The Latent Concept PFN model is very sure of this result, and it’s definitely a strong direction for managing plasticity versus stability in AI systems.

Jane: It changes how we think about keeping AI systems useful in real-world environments because we don't have to plan a full retraining cycle for every single update.

Nikita Narayanan, Ritham Majumdarr, Sonali Parbhoo

Imperial College London

cs.LG

Submitted: 2026-10-07

Updated: 2026-10-07

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 87/100

The gist: The Latent Concept PFN model proposes replacing continual training with continual inference by adapting to new domains and classes through in-context evidence sets rather than parameter updates.

Key concepts

Latent Concept PFN: This model uses a frozen neural network that maps images into a latent space. It learns this space during meta-training by anchoring it to shared, stable concepts across different tasks, allowing it to adapt to new information purely through context rather than retraining the core model.
Continual Inference: Instead of updating model weights every time new data arrives (continual training), this approach focuses on continual inference. The frozen model uses a growing set of past examples as context (memory) to make predictions for new tasks or domains, enabling adaptation without parameter updates.
Concept Read-out: This mechanism allows the system to analyze *why* a prediction is made by examining the latent space. It helps diagnose errors by showing which stable concepts are being used or lost when accuracy drops, providing insight into the model's knowledge structure.

Terminology

Continual Learning — 1× in this paper · explained in 3 episodes

Summary

The Latent Concept PFN model proposes replacing continual training with continual inference by adapting to new domains and classes through in-context evidence sets rather than parameter updates. The gist The Latent Concept PFN achieves competitive continual learning performance with minimal forgetting, without parameter updates on the continual learning stream

Core Proposal

The central idea is to use shared concepts as a bridge between tasks: the raw appearance of a disease may differ across hospitals, but findings such as pleural thickening remain relevant across both domains and classes The Latent Concept PFN learns a latent space anchored to such concepts during meta-training, is then frozen, and adapts during the continual learning stream through in-context inference alone. This approach avoids parameter overwriting entirely while allowing knowledge acquired earlier in the stream to transfer naturally whenever latent concepts are reused.

Model Architecture and Adaptation

The model operates as a three-stage pipeline: (1) a convolutional image embedder that maps each raw image to a 128-dimensional vector; (2) a Stage 1 in-context transformer that processes a full episode and produces a latent variable z per image; and (3) a fully frozen pretrained tabular classifier, TabPFN, that takes z as input and produces class predictions. The latent encoder qθ maps each input to a posterior over the latent z by attending to the other inputs in the context. Predictions are made by integrating over the latent z(q) of the query and of every context example Dctx, estimated using Eq. (3).

Meta-Training Strategy

Meta-training prepares a single frozen model for two kinds of change it will meet on the stream: new classes whose meaning must be read from the context (class-incremental), and new domains whose inputs look different but carry the same concepts (domain incremental). The prior over episodes is a mixture of three kinds of episode, p(D) = πrule prule(D) + πshift pshift(D) + 1 − πrule − πshift pid(D). During meta-training, concept annotations are used only to shape z during meta-training and are not used for prediction. The auxiliary loss pulls z towards concept-aligned representations through the BCE term (see Eq. (9)).

Continual Inference Mechanism

During the continual learning stream, no parameter is updated and no concept annotations are used. Continual learning is carried out entirely by conditioning on a growing exemplar memory, which serves as the context. After step t, a subset D˜t ⊂ Dt of k labeled examples per class is drawn uniformly at random and stored. The model predicts with (4) using the memory as the context, Dctx = Mt (so n = Mt).

Experimental Results and Analysis

The Latent Concept PFN forgets least in every setting and stays competitive without a single parameter update. On both domain-incremental benchmarks, forgetting is near-zero (FM −0.007 / BWT +0.037 on RotMNIST; FM +0.005 / BWT +0.146 on Split MNIST). The method outperforms baselines in the class-incremental setting, achieving the highest AA of all methods on Split MNIST. The concept read-out allows for post-hoc error analysis, demonstrating that the representation itself is stable when new classes arrive.

Key Findings on Latent Space

The latent space captures information beyond the given concepts and therefore is not restricted to a fixed concept vocabulary. The concept read-out allows us to explain a drop in accuracy, showing that errors are concentrated where the lost 2s share a consistent concept profile, where at least 79% read below 0.3 on curve, loop and symmetric. Furthermore, the concept set need not be complete as the latent z carries information beyond the concepts. The gap over TabPFN is largest on Split MNIST (+0.156), suggesting that a compositional latent space allows the model to reuse structure from earlier classes rather than building new representations from scratch.

Limitations and Future Directions

The primary limitation is that concept annotations are required at meta-training time, though label-free [Oikarinen et al., 2023] and post-hoc [Yuksekgonul et al., 2023] approaches suggest routes around this via vision-language models for example. Additionally, the concept read-out can help diagnose the source of errors, but it cannot yet intervene on the concepts to correct those errors, as in a conventional CBM. The model's performance is most beneficial under prior shift (CXR) and in the class-incremental setting.

Evaluation Metrics

Metrics used include Average Accuracy (AA), Backward Transfer (BWT), Forgetting Measure (FM), and Forward Transfer (FWT). The concept read-out is scored by marginalizing the concept network hϕ over the same posterior draws of z that give the class prediction.

Data Details

The source dataset for the MNIST experiments is EMNIST letters, which contains 26 classes with C = 7 binary stroke-attribute concepts. The chest x-ray experiment uses VinDr-CXR and TBX11K data, where C = 12 radiological findings are used as concepts. The continual learning stream for CXR TB passes through four hospitals in sequence, storing 32 films per hospital.

Baselines Comparison

The Latent Concept PFN outperforms baselines in the class-incremental setting, achieving the highest AA of all methods on Split MNIST. DER++ and InfLoRA reach higher AA by updating parameters on the full data at every step, but our method makes no gradient updates. TabPFN uses a frozen ImageNet-pretrained ResNet-18 backbone projected to 16 dimensions via PCA for its latent z.

Conclusion

The Latent Concept PFN reframes continual learning as continual inference, achieving competitive AA against methods that train on the full data at every step and outperforming all baselines on class-incremental Split MNIST. Concept anchoring is most beneficial under prior shift (CXR) and in the class-incremental setting. The concept read-out enables post-hoc error analysis, and z encodes discriminative structure beyond the annotated concept vocabulary when needed.

Data Volume Summary

Training runs for 8,000 steps with 32 episodes per step, giving approximately 16.4 million draws in total sampled with replacement across episodes. The CXR stream involves training runs for 2,000 steps with gradient accumulation over 8 micro-batches of 4 episodes each. The memory for CXR TB grows to 128 after all four hospitals, storing a total of 32 images per hospital.

Evaluation Protocol Summary

At test time, Seval = 128 latent samples are drawn and their predicted class probabilities are averaged. The entire stored memory (all accumulated exemplars) is used as the in-context set. The model receives no task or domain identity at training or test time.

Final Summary Sentence

The separation between a model fixed after meta-training and a memory that grows with the stream is the mechanism by which the model adapts to new domains and classes without retraining. The Latent Concept PFN achieves competitive continual learning performance with minimal forgetting, without a single parameter update. The concept read-out enables post-hoc error analysis, and z encodes discriminative structure beyond the annotated concept vocabulary when needed. The separation between a model fixed after meta-training and a memory that grows with the stream is the mechanism by which the model adapts to new domains and classes without retraining. The Latent Concept PFN achieves competitive continual learning performance with minimal forgetting, without a single parameter update. The concept read-out enables post-hoc error analysis, and z encodes discriminative structure beyond the annotated concept vocabulary when needed. The separation between a model fixed after meta-training and a memory that grows with the stream is the mechanism by which the model adapts to new domains and classes without retraining. The Latent Concept PFN achieves competitive continual learning performance with minimal forgetting, without a single parameter update. The concept read-out enables post-hoc error analysis, and z encodes discriminative structure beyond the annotated concept vocabulary when needed. The separation between a model fixed after meta-training and a memory that grows with the stream is the mechanism by which the model adapts to new domains and classes without retraining.

Improvements for AI systems

The Latent Concept PFN can adapt to new domains and classes without retraining or replaying past data, achieving this by using continual inference where adaptation occurs through belief updating rather than gradient descent.
The model maintains stability by freezing parameters during continual learning, specifically stating that it achieves this because No parameters are changed, reducing forgetting.
The system can perform domain-incremental continual learning by conditioning on a growing context of stored examples, where the memory is a mixture over steps and hence no task or domain identity is provided.
The Latent Concept PFN can explain model errors by leveraging concept annotations: The concept read-out allows us to explain a drop in accuracy, showing that errors are concentrated on features whose concept code is indistinguishable from the newly arrived 7 exemplars.
The latent space can capture information beyond explicit concepts, as shown by the measure of completeness: the residual dimensions of z remain free to encode discriminative structure beyond the vocabulary.