Freeze, Diffuse, Decode: Task-Aware Adaptation of Transformer Embeddings for Antimicrobial Peptide Design
cs.LG
Submitted: 2025-11-28
Updated: 2026-09-02
Comments: 16 pages, 4 figures
License: http://creativecommons.org/licenses/by/4.0/
The gist: Pretrained transformers provide rich, general-purpose embeddings, which are transferred to downstream tasks.
Terminology
Abstract
Pretrained transformers provide rich, general-purpose embeddings, which are transferred to downstream tasks. However, current transfer strategies: fine-tuning and probing, either distort the pretrained geometric structure of the embeddings or lack sufficient expressivity to capture task-relevant signals. These issues become even more pronounced when supervised data are scarce. Here, we introduce Freeze, Diffuse, Decode (FDD), a novel diffusion-based framework that adapts pre-trained embeddings to downstream tasks while preserving their underlying geometric structure. FDD propagates supervised signal along the intrinsic manifold of frozen embeddings, enabling a geometry-aware adaptation of the embedding space. Applied to antimicrobial peptide design, FDD yields low-dimensional, predictive, and interpretable representations that support property prediction, retrieval, and latent-space interpolation.
Sources
- The Evolution of Out-of-Distribution Robustness Throughout Fine-Tuning
- Random Forest Autoencoders for Guided Representation Learning
- Neural Ordinary Differential Equations
- ChemBERTa: Large-Scale Self-Supervised Pretraining for Molecular Property Prediction
- Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distances
- Enhancing Molecular Property Prediction with Auxiliary Learning and Task-Specific Adaptation
- A Strong Baseline for Molecular Few-Shot Learning
- Synergistic Benefits of Joint Molecule Generation and Property Prediction
- Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution
- Optimal Transport losses and Sinkhorn algorithm with general convex regularization
- Benchmarking Pretrained Molecular Embedding Models For Molecular Representation Learning
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks