Protein Language Models Fail To Capture Contact Predictions Under Invariant Relabeling
summary
The gist
Protein language models (pLMs) produce highly accurate contact maps from single-sequence inputs, and these maps collapse under arbitrary relabeling that leaves classical alignment-based methods…
In short
Protein Language Models (pLMs) generate accurate contact maps from single sequences, but these maps collapse when amino acid letters are arbitrarily relabeled. Classical methods like Potts models remain unchanged because they rely on column statistics. This reveals that pLMs do not learn pure coevolutionary statistics; instead, they depend on explicit residue identities or 'alphabet anchoring' to compute contacts.
Key concepts
- Global Bijective Relabeling
- This is a transformation where every unique amino acid letter in the protein sequence is swapped with another unique letter. Crucially, this process permutes the letters but leaves all alignment-level information, column entropies, and pairwise edit distances completely unchanged. It serves as a diagnostic tool to test if a model relies on specific residue identities.
- Pseudolikelihood Potts Model
- This is a classical statistical method used to predict protein contacts based on sequence alignments. The paper proves that this model is invariant under global relabeling because its objective function remains identical and the parameter space mapping is bijective. It serves as the benchmark for what contact prediction should look like if it relies only on general statistical properties.
- Alphabet Anchoring
- This phenomenon describes how pLMs fail when their alphabet is relabeled, even though classical models succeed. Instead of learning universal coevolutionary rules, the pLM implicitly learns to condition its predictions on the specific chemical identity of each residue. This 'anchoring' means the model relies on knowing which specific amino acid is at a position rather than just general evolutionary trends.
- Column Statistics
- These are summary measures derived from an alignment that describe the statistical properties of residues at specific positions across different sequences. The paper shows that estimators relying solely on these statistics, such as the inverse-covariance estimator, are provably invariant under relabeling because they only depend on the relative arrangement and frequency of residues, not their specific labels.
Terminology used across episodes
This episode discusses
- Protein Language Models Fail To Capture Contact Predictions Under Invariant Relabeling · Paper Radio
- Scaling Laws for Neural Language Models
- Protein contacts are already in the attention: a single-forward-pass alternative to the Categorical Jacobian
- BERTology Meets Biology: Interpreting Attention in Protein Language Models
The paper
Protein Language Models Fail To Capture Contact Predictions Under Invariant Relabeling · Read on arXiv
Paarth Batra, Aryan Keluskar
The Biodesign Institute
Protein language models (pLMs) produce highly accurate contact maps from single-sequence inputs. People often take this accuracy as proof that these models implicitly learn invariant coevolutionary statistics, extracting the same signals as classical alignment-based methods like Potts. In this work, we introduce a diagnostic framework around a global bijective relabeling of the amino-acid alphabet. This transformation permutes residue letters while leaving all alignment-level information, column entropies, and pairwise edit distances unchanged. Pseudolikelihood Potts models and inverse-covariance estimators remain invariant under this transformation, but pLMs fail the test. Across multiple architectures (ESM-2, MSA Transformer, AMPLIFY) and model scales (8M to 3B parameters), pLM contact maps collapse under arbitrary relabeling. Using Hamming distance controls and biochemical grouping interventions, we attempt to isolate what likely causes this degradation and find that these models rely on explicit, canonical residue identities (``alphabet anchoring'') rather than pure coevolutionary statistics. Increasing scale makes models more responsive to biochemical classes, but their performance still degrades under relabeling. Our findings show a gap between how pLMs and classical coevolutionary models compute contacts. More broadly, our work shows how diagnostic perturbations that leave classical models provably unchanged can expose hidden representation biases and prevent over-reliance on neural black boxes in biological modeling.
Transcript
Introduction to the show: ident: Genomics Radio. Generated commentary on the latest computational biology and genomics papers.
Ines: I'm Ines.
Marcus: And I'm Marcus, here with Yuki.
Ines: We're looking at the paper: "Protein Language Models Fail To Capture Contact Predictions Under Invariant Relabeling".
Marcus: Here's the gist of the paper. Protein language models (pLMs) produce highly accurate contact maps from single-sequence inputs, and these maps collapse under arbitrary relabeling that leaves classical alignment-based methods invariant The authors introduce a diagnostic framework using global bijective relabeling of the amino-acid alphabet to test whether pLM contact readouts rely on explicit residue identities rather than pure coevolutionary statistics This work demonstrates that while classical estimators like Potts models remain invariant under this transformation, pLMs fail the test because they rely on "explicit, canonical residue identities" or "alphabet anchoring" Testing Invariance of Classical Estimators The authors establish that any estimator reading only column statistics must be invariant under a global bijection because column entropies and the true contact map are unchanged by construction They prove this invariance for the pseudolikelihood Potts model and, in closed form, for the inverse-covariance estimator Specifically, Proposition 2 shows that the inverse-covariance estimator remains accurate under relabeling because it commutes with the permutation matrix transformation Failure Modes of Protein Language Models Every pLM tested degrades when its alphabet is globally relabeled, showing that they do not compute contacts solely from column statistics The failure is not tied to a single architecture or tokenizer, as the degradation replicates across ESM-2, MSA Transformer, and AMPLIFY Furthermore, the degradation costs accuracy; for ESM-2 650M on ubiquitin, precision drops from 2.33× the Potts precision to 0.17× under relabeling Distinguishing Identity from Edit Distance The study isolates the cause of degradation by using three regimes—random, in-class, and cross-class permutations—to separate edit distance from biochemical class The authors introduce a matched-Hamming control to explicitly separate edit distance from residue identity by ensuring all arms mutate the same number of residues at the same positions This reveals that models rely on "residue identity conditioned pattern matching" rather than continuous distance metrics Implications for Biological Modeling The findings suggest a gap between how pLMs and classical coevolutionary models compute contacts, showing that high contact precision cannot be cited as proof that a model computes contacts through coevolution The phenomenon is termed "alphabet anchoring," where the readout holds when residues are swapped within biochemical classes but collapses when swaps cross class boundaries This diagnostic perturbation exposes hidden representation biases and prevents over-reliance on neural black boxes in biological modeling The authors conclude that scaling does not close the gap, as larger models still degrade under arbitrary relabeling Limitations of the Study The scope is limited to a relatively small number of proteins and specific readouts, meaning the results are necessary but not sufficient evidence for coevolutionary inference The authors note that invariance under alphabet relabeling is a necessary condition for a readout to be a pure function of column statistics, but it is not sufficient to prove learned coevolution The study also did not test retrained or fine-tuned models, as the claim concerns public pre-trained checkpoints The work releases all code and data used for reproducibility References
2: Thomas M.
Ines: Cover and Joy A.
Marcus: Thomas.
Ines: Elements of Information Theory
3: Margaret O.
Paper summary: Ines: So we're looking at this paper right now, "Protein Language Models Fail To Capture Contact Predictions Under Invariant Relabeling." Basically, they take these protein language models and show that even though they are super accurate at predicting contacts from a single sequence, those contacts aren't actually based on the deep coevolutionary statistics people usually think they learn.
Marcus: Right. The core idea here is this diagnostic framework involving a global bijective relabeling of the amino acid alphabet. They permute all the letters—say, swapping 'alanine' with 'glycine' everywhere—but they claim this transformation leaves all the alignment statistics, column entropies, and pairwise edit distances completely unchanged.
Yuki: That sounds like a test designed to see if you can separate learned patterns from raw sequence identity. If an estimator only reads those column statistics, it should return the same contact map regardless of how you rename the letters, right?
Ines: Exactly. They show that classical methods, like the pseudolikelihood Potts model and even the inverse-covariance estimator, are totally invariant under this relabeling transformation because they only depend on those column statistics or edit distances <ref:2611.16881#pg2>.
Marcus: But the protein language models fail that test across multiple architectures like ESM two point zero, the MSA transformer, and AMPLIFY. They collapse their contact maps when you apply this arbitrary relabeling <ref:2611.16881#pg2>.
Yuki: So what does that mean for the biology? If these models aren't really learning those coevolutionary statistics, then the high accuracy we see isn't actually proof of that biological process being modeled <ref:2611.16881#pg3>.
Ines: It suggests they are instead relying on something more direct, something that anchors them to specific residue identities <ref:2611.16881#pg3>. The paper calls this "alphabet anchoring," meaning the model is mapping contacts based on what the specific amino acid letter *is*, rather than just the statistical relationship between positions <ref:2611.16881#pg3>.
Marcus: They try to isolate this by looking at different types of perturbations, using random, in-class, and cross-class permutations to separate simple edit distance from the actual biochemical class of residue <ref:2611.16881#pg2>. They even used a matched hamming control to make sure all the arms mutated the same number of residues at the same positions.
Yuki: That's interesting because it moves beyond just looking at accuracy and tries to pinpoint *why* it breaks, which is what we need for real biological understanding <ref:2611.16881#pg2>. What does that mean for how we interpret these models in a population context?
Ines: It means the way pLMs compute contacts is fundamentally different from how classical coevolutionary models compute them <ref:2611.16881#pg3>. The shared structure of the final contact score pipeline, whether it's the Potts tensor or the categorical Jacobian, is what makes this comparison possible <ref:2611.16881#pg3>.
Marcus: The paper shows that this gap between how these models compute contacts and how classical methods do is real. It also points out that increasing model size, going from eight million to three billion parameters, doesn't solve the problem; in fact, larger models are more responsive to biochemical classes but still degrade under relabeling <ref:2611.16881#pg2>.
Yuki: That tells us that scaling up the model isn't the answer for capturing true coevolutionary signals if the underlying mechanism is based on identity matching rather than pure statistical inference <ref:2611.16881#pg2>. So, what’s the implication for building predictive tools?
Ines: The implication is that we can use these diagnostic perturbations to expose hidden representation biases in these neural black boxes and stop over-relying on them without needing a full theory of coevolution <ref:2611.16881#pg3>. It’s about using the math itself as a probe for what the model is actually learning about the sequence structure <ref:2611.16881#pg3>.
Marcus: So to wrap up, this paper "Protein Language Models Fail To Capture Contact Predictions Under Invariant Relabeling" shows that high contact precision from protein language models doesn't automatically mean they are computing contacts through coevolution <ref:2611.16881#pg3>. It proves they rely on explicit residue identities or alphabet anchoring, which is a different mechanism than what classical methods use to separate direct coevolution from indirect correlation <ref:2611.16881#pg3>.
Yuki: And for us looking at the bigger picture, it means that when we trust these models for biological inference, we have to be careful about whether they're capturing deep evolutionary patterns or just surface-level sequence matching <ref:2611.16881#pg2>. That’s a big distinction in how you interpret the data coming out of these large language models.
Conclusion: Ines: So we're wrapping up this deep dive on these protein language models and their ability to predict contacts from a single sequence.
Marcus: Yeah, this paper, "Protein Language Models Fail To Capture Contact Predictions Under Invariant Relabeling," it basically shows that just because a model gets high accuracy on its predictions isn't proof that it's actually learning the deep evolutionary statistics we think it is.
Yuki: So if those models aren't capturing the coevolution, what are they really doing instead? Are they just memorizing patterns?
Ines: They seem to be relying on something more rigid, something like explicit residue identities or what the authors call "alphabet anchoring," rather than just looking at statistical relationships.
Marcus: That means when you swap out one amino acid for another within the same family, but cross a different biochemical class, the model breaks completely. It only works if you keep the letters in their original biochemical neighborhoods.
Yuki: So this suggests there’s a real difference between how these neural models are making predictions and how classical methods based on statistical inference actually work.
Ines: Exactly. The numbers show that when you test them with random relabeling, the accuracy drops from being very high to almost nothing, while classical estimators like the Potts model stay exactly the same because they're built around those column statistics.
Marcus: The authors point out that even as these models get bigger—we’re talking eight million to three billion parameters—they still degrade under this arbitrary relabeling test. Scaling up didn't fix their fundamental issue.
Yuki: That’s a pretty sobering thought for population genetics, meaning that just because we can train a massive model doesn't automatically mean it has captured the underlying biological truth about how proteins actually interact over evolutionary time.
Ines: It really highlights this gap between how these pLMs compute contacts and how classical coevolutionary models do, which opens up a new way to look at what these neural networks are actually learning.
Marcus: And this whole concept of "alphabet anchoring" could be the key to figuring out why we see such high precision in protein language models right now.
More episodes
- 2610.33881-Sensorimotor behavior from an untrained brain-and-nerve-cord connectome
- 2610.40344-Morphological Features of Murine Pulmonary Arterial Networks for Diagnosing Vascular Disease
- Daily Summary for 2026-11-19
- 2609.10193-Are You Learning Biological Signal or Shortcuts? Auditing and Mitigating Bias in Protein-Protein Interaction Datasets
- 2610.38236-Generative discovery of chemical reaction networks
- 2507.11176-An Interpretable AI framework Quantifying Traditional Chinese Medicine Principles Towards Enhancing and Integrating with Modern Biomedicine
- The day in conversation for 2026-11-19
- Daily Summary for 2026-11-19
- 2504.03590-Towards a Unified Framework for Determining Conformational Ensembles of Disordered Proteins
- 2603.15217-A multiscale discrete-to-continuum framework for structured population models