Evolutionary Curriculum Learning Improves Biological Sequence Modeling

Evolutionary Curriculum Learning (ECL) is a training strategy for variational autoencoders (VAEs) trained on multiple sequence alignments (MSAs) of biological sequences. Standard biological VAE training treats all sequences as exchangeable, ignoring the evolutionary structure that organizes homologous sequences from evolutionarily close to highly divergent. ECL exploits this structure by progressively exposing the model to sequences of increasing evolutionary distance from sampled anchors, following a power-law expansion schedule. The method requires no architectural modifications, only pairwise distances computed once during preprocessing.

For proteins (EVE), pairwise distances are estimated under the WAG substitution model with a four-category discrete-Gamma correction (shape parameter α = 1.0), yielding WAG+Γ distances. For RNA (RfamGen), distances are computed in the covariance model (CM) feature space as the sum of normalized squared l2 distances across the three CM components (transition rules, single-strand emissions, and base-pair emissions), which recovers the per-position Hamming rate.

At each training step, ECL samples an anchor sequence from the full training set with probability proportional to phylogenetic sequence weights πs. Then, B−1 neighbor sequences are sampled (also proportional to πs) from the k(t)-nearest neighbors of the anchor. The neighborhood size k(t) expands over training via a schedule that interpolates log-linearly from an initial size k0 to the full training set N−1, with exponent γ = 2.0 used throughout. At t=1, k=k0; at t=T, k=N−1, so all sequences are reachable. Because k grows multiplicatively rather than additively, early training is spent almost entirely within a tight local neighborhood, then expands rapidly in the latter half of training.

ECL was applied to two architecturally distinct VAE models and two biological domains: protein variant effect prediction with EVE and RNA family sequence generation with RfamGen. Every configuration was trained with five random seeds (0–4), and each ECL variant was compared against the seed-matched baseline so comparisons are paired.

For protein variant effect prediction, EVE was trained on three disease-relevant proteins: p53 (n=3,666 MSA sequences), PTEN (n=1,195), and SCN5A (n=11,846). ECL converges to lower validation negative ELBO and reconstruction loss on all three proteins. On p53, ECL raises mean ClinVar classification AUROC from 0.9814 to 0.9889 (+0.0075, 95% CI [−0.0012, +0.0161], t-test p = 0.075), winning on 4 of 5 seeds. On PTEN, ECL attains 1.0000 on every seed, while the baseline averages 0.9054 with a very wide interval: four baseline seeds score ≥ 0.99 but one collapses to 0.5405. The improvement on PTEN is best described as a gain in stability rather than a large shift in central tendency, and the paired difference is not significant (p = 0.36) precisely because the baseline variance is so large. SCN5A is omitted from AUROC reporting because after intersecting with the modeled MSA positions its labeled set contains 23 variants, all pathogenic, leaving AUROC undefined. A cross-validated position-only classifier (k-NN on residue index) reaches AUROC 0.837 on p53 and 0.423 on PTEN, both far below the VAE scores, confirming that the models capture substitution-level signal rather than a spatial artifact.

For RNA family sequence generation, RfamGen was trained on three Rfam families: RF00169 (n=6,168), RF00234 (n=900), and RF00638 (n=2,999). After training, 1,000 sequences per model were sampled and scored with the ground-truth covariance model to compute bit scores, where higher values indicate sequences that better resemble natural RNA family members. Averaged over seeds, ECL yields higher mean bit scores on all three families: 86.0 vs. 83.0 for RF00169 (+3.7%, 5/5 seeds, p = 0.077), 150.2 vs. 139.6 for RF00234 (+7.5%, 3/5, p = 0.33), and 153.7 vs. 152.3 for RF00638 (+0.9%, 4/5, p = 0.77). Individually these per-family effects are underpowered at n = 5. Aggregating the 15 seed-matched comparisons, ECL wins 12 of 15 runs, which an exact sign test rejects at p = 0.035. The unit of inference for this test is the training run, not the RNA family: it supports the claim that ECL reliably improves training runs on these three families, not that it improves RNA families in general. With only three families, a family-level test cannot establish significance even if every family improves (a two-sided sign test over three families floors at p = 0.25).

Ablation experiments isolate the contribution of the curriculum. A fixed-k variant retains anchor-based local sampling but holds k(t) = k0 for the entire run. On p53, fixed-k is clearly harmful: mean AUROC falls to 0.9483, −0.0332 below the baseline (95% CI [−0.0382, −0.0281], p = 0.0001), losing on all five seeds. Training only on tight neighborhoods starves the model of the global variation it needs, and the benefit of ECL depends on the expansion rather than on locality alone. On PTEN, fixed-k also reaches 1.000 AUROC, but this is uninformative because with only two benign variants the metric is saturated. Across RNA families fixed-k improves the pooled bit score by only 2.3% (7/15 runs) compared to 5.0% (12/15) for full ECL.

For RfamGen, ECL is naturally paired with a non-cyclic KL annealing schedule, so all four combinations of baseline, ECL sampling × cyclic, non-cyclic annealing were run. Changing only the annealing (baseline + non-cyclic) does not reproduce ECL: pooled across families it lowers the mean bit score by 5.0%, and it is substantially harmful on RF00234 (−11.5%) and RF00638 (−10.6%) while helping only RF00169 (+3.4%). Conversely, ECL combined with the original cyclic schedule still improves over the baseline (+5.7% pooled). The curriculum sampler, not the annealing schedule, is therefore responsible for the improvement. ECL helps under either annealing schedule, but the non-cyclic schedule is the more consistent of the two (12/15 vs. 8/15 winning runs).

Sensitivity analysis sweeps the initial neighborhood k0 and the schedule exponent γ around their defaults. On p53 every setting stays at or above the baseline (k0 ∈ 255, 512, 1024 → 0.9889, 0.9846, 0.9892; γ ∈ 1, 2, 4 → 0.9893, 0.9889, 0.9830), and across RNA families the pooled improvement varies only between +4.7% and +6.1% over the same grid. ECL is therefore not delicately tuned. The one systematic failure is informative: on PTEN, k0 = 1024 drops mean AUROC to 0.7541 and wins on only 1 of 5 seeds. PTEN’s training set has N−1 = 1060, so k0 = 1024 makes the initial neighborhood nearly the entire alignment: the curriculum is effectively removed and sampling is global from the first step. The same k0 is harmless on p53, whose alignment is roughly three times larger and for which 1024 is still a genuinely local neighborhood.

ECL adds no per-step training overhead: sampling an anchor and its neighbors is O(B) per step, exactly as for weighted uniform sampling. The only additional cost is the pairwise distance matrix, computed once and cached, then reused across every run, seed, and hyperparameter setting. This preprocessing is O(N2) in the number of sequences for both protein and RNA modalities. Because the curriculum consumes only ordered neighbor lists rather than exact distances, any monotone proxy would suffice; identifying cheaper such proxies is left to future work.

ECL is a training-strategy intervention rather than a new model class, so the comparison reported is baseline-EVE versus ECL-EVE with identical architecture, data splits, optimizer, and compute budget. ECL is orthogonal to, and composable with, any MSA-based generative predictor: it changes only the order in which training sequences are visited.

Several limitations are acknowledged. Effect sizes are modest relative to seed variance: with five seeds per configuration, most individual comparisons do not reach p < 0.05, and the strongest evidence is the consistency of the direction together with the ablations rather than any single headline number. The aggregate RNA statistic (12 of 15 runs, p = 0.035) treats the training run as the unit of inference; because seeds within a family share an alignment, it does not by itself license generalization to unseen RNA families, and family-level tests over three families cannot achieve significance. The clinical evaluation sets are also small and imbalanced—PTEN retains two benign variants and SCN5A none once labels are intersected with the modeled MSA positions—so protein AUROC is a coarse instrument here and saturates easily. The evaluation covers three proteins and three RNA families, which is too few to support strong claims of generality. Finally, the curriculum requires an O(N2) preprocessing step.

In conclusion, across two biological modalities—protein and RNA—and two architecturally distinct models, ECL improves downstream task performance in a consistent direction: variant classification AUROC for disease prediction and CM bit scores for RNA generation quality. Evolutionary distance is therefore a useful inductive bias for ordering the training curriculum in biological sequence modeling.

Improvements to AI Systems:

1. Evolutionary-Ordered Training Scheduler – Replace uniform random sampling in sequence models with a curriculum that progressively expands from evolutionarily close to distant sequences, using a power-law expansion schedule (γ=2.0). The improved system learns global sequence constraints without being starved by local-only data, leading to more robust generative models.

2. Anchor-Based Local Sampling with Dynamic Neighborhood – At each step, sample an anchor sequence (weighted by phylogenetic importance) and draw neighbors from a growing k-nearest-neighbor set. This forces the model to first master fine-grained local variation, then generalize to broader evolutionary divergence. The improved system achieves better convergence (lower validation loss) and higher downstream accuracy.

3. Task-Agnostic Curriculum for Biological Sequence Models – Apply the same training strategy to any VAE or generative model over MSAs (protein or RNA). The improved system can be used for variant effect prediction (higher AUROC, e.g., +0.0075 on p53, perfect stability on PTEN) and RNA family generation (higher CM bit scores, +3–7% across families).

4. Stability Enhancement for Imbalanced Clinical Datasets – Use the curriculum to prevent catastrophic collapse on small, skewed datasets (e.g., PTEN with only 2 benign variants). The improved system maintains high performance across all seeds, avoiding single-seed failures (baseline had one seed drop to 0.54 AUROC; ECL achieves 1.0 on all seeds).

5. No-Architecture-Change Drop-in Training Module – Integrate ECL as a plug-in sampler that requires only a precomputed pairwise distance matrix (O(N2) one-time cost). The improved system gains performance without modifying model architecture, inference, or compute budget per step.

6. Robustness to Hyperparameter Choice – Use the curriculum’s insensitivity to initial neighborhood size and expansion exponent (performance varies <1% across tested ranges). The improved system avoids delicate tuning, making it practical for deployment across diverse sequence families.

7. Composable with Existing Annealing Schedules – Combine ECL with any KL annealing scheme (cyclic or non-cyclic) without losing benefits. The improved system can be layered onto existing training pipelines, yielding consistent gains (12/15 winning runs pooled across families).

8. Prevention of Over-Localization – Avoid fixed small neighborhoods (which harm performance, e.g., −3.3% AUROC on p53) by ensuring the curriculum expands to full dataset. The improved system prevents mode collapse from overfitting to closely related sequences.

What the Improved AI System Can Do:

• Predict pathogenic vs. benign protein variants with higher and more stable accuracy, even on tiny clinical datasets (e.g., perfect AUROC on PTEN across all seeds).

• Generate novel RNA sequences that score closer to natural family members (higher covariance model bit scores), improving functional plausibility.

• Train on any MSA-based generative task (proteins, RNA, potentially DNA) with faster convergence and better final performance, without architectural changes.

• Operate reliably across different dataset sizes (from 900 to 12,000 sequences) and evolutionary depths, with no extra per-step computational cost.

• Provide a drop-in replacement for standard sampling in existing VAE training loops, requiring only a one-time distance matrix computation.