Protein Language Models Fail To Capture Contact Predictions Under Invariant Relabeling

arXiv:2611.16881 · q-bio.QM, cs.AI · Submitted 2026-11-12 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Genomics Radio. Generated commentary on the latest computational biology and genomics papers.

Ines: I'm Ines.

Marcus: And I'm Marcus, here with Yuki.

Ines: We're looking at the paper: "Protein Language Models Fail To Capture Contact Predictions Under Invariant Relabeling".

Marcus: Here's the gist of the paper. Protein language models (pLMs) produce highly accurate contact maps from single-sequence inputs, and these maps collapse under arbitrary relabeling that leaves classical alignment-based methods invariant The authors introduce a diagnostic framework using global bijective relabeling of the amino-acid alphabet to test whether pLM contact readouts rely on explicit residue identities rather than pure coevolutionary statistics This work demonstrates that while classical estimators like Potts models remain invariant under this transformation, pLMs fail the test because they rely on "explicit, canonical residue identities" or "alphabet anchoring" Testing Invariance of Classical Estimators The authors establish that any estimator reading only column statistics must be invariant under a global bijection because column entropies and the true contact map are unchanged by construction They prove this invariance for the pseudolikelihood Potts model and, in closed form, for the inverse-covariance estimator Specifically, Proposition 2 shows that the inverse-covariance estimator remains accurate under relabeling because it commutes with the permutation matrix transformation Failure Modes of Protein Language Models Every pLM tested degrades when its alphabet is globally relabeled, showing that they do not compute contacts solely from column statistics The failure is not tied to a single architecture or tokenizer, as the degradation replicates across ESM-2, MSA Transformer, and AMPLIFY Furthermore, the degradation costs accuracy; for ESM-2 650M on ubiquitin, precision drops from 2.33× the Potts precision to 0.17× under relabeling Distinguishing Identity from Edit Distance The study isolates the cause of degradation by using three regimes—random, in-class, and cross-class permutations—to separate edit distance from biochemical class The authors introduce a matched-Hamming control to explicitly separate edit distance from residue identity by ensuring all arms mutate the same number of residues at the same positions This reveals that models rely on "residue identity conditioned pattern matching" rather than continuous distance metrics Implications for Biological Modeling The findings suggest a gap between how pLMs and classical coevolutionary models compute contacts, showing that high contact precision cannot be cited as proof that a model computes contacts through coevolution The phenomenon is termed "alphabet anchoring," where the readout holds when residues are swapped within biochemical classes but collapses when swaps cross class boundaries This diagnostic perturbation exposes hidden representation biases and prevents over-reliance on neural black boxes in biological modeling The authors conclude that scaling does not close the gap, as larger models still degrade under arbitrary relabeling Limitations of the Study The scope is limited to a relatively small number of proteins and specific readouts, meaning the results are necessary but not sufficient evidence for coevolutionary inference The authors note that invariance under alphabet relabeling is a necessary condition for a readout to be a pure function of column statistics, but it is not sufficient to prove learned coevolution The study also did not test retrained or fine-tuned models, as the claim concerns public pre-trained checkpoints The work releases all code and data used for reproducibility References

2: Thomas M.

Ines: Cover and Joy A.

Marcus: Thomas.

Ines: Elements of Information Theory

3: Margaret O.

Paper summary: Ines: So we're looking at this paper right now, "Protein Language Models Fail To Capture Contact Predictions Under Invariant Relabeling." Basically, they take these protein language models and show that even though they are super accurate at predicting contacts from a single sequence, those contacts aren't actually based on the deep coevolutionary statistics people usually think they learn.

Marcus: Right. The core idea here is this diagnostic framework involving a global bijective relabeling of the amino acid alphabet. They permute all the letters—say, swapping 'alanine' with 'glycine' everywhere—but they claim this transformation leaves all the alignment statistics, column entropies, and pairwise edit distances completely unchanged.

Yuki: That sounds like a test designed to see if you can separate learned patterns from raw sequence identity. If an estimator only reads those column statistics, it should return the same contact map regardless of how you rename the letters, right?

Ines: Exactly. They show that classical methods, like the pseudolikelihood Potts model and even the inverse-covariance estimator, are totally invariant under this relabeling transformation because they only depend on those column statistics or edit distances <ref:2611.16881#pg2>.

Marcus: But the protein language models fail that test across multiple architectures like ESM two point zero, the MSA transformer, and AMPLIFY. They collapse their contact maps when you apply this arbitrary relabeling <ref:2611.16881#pg2>.

Yuki: So what does that mean for the biology? If these models aren't really learning those coevolutionary statistics, then the high accuracy we see isn't actually proof of that biological process being modeled <ref:2611.16881#pg3>.

Ines: It suggests they are instead relying on something more direct, something that anchors them to specific residue identities <ref:2611.16881#pg3>. The paper calls this "alphabet anchoring," meaning the model is mapping contacts based on what the specific amino acid letter *is*, rather than just the statistical relationship between positions <ref:2611.16881#pg3>.

Marcus: They try to isolate this by looking at different types of perturbations, using random, in-class, and cross-class permutations to separate simple edit distance from the actual biochemical class of residue <ref:2611.16881#pg2>. They even used a matched hamming control to make sure all the arms mutated the same number of residues at the same positions.

Yuki: That's interesting because it moves beyond just looking at accuracy and tries to pinpoint *why* it breaks, which is what we need for real biological understanding <ref:2611.16881#pg2>. What does that mean for how we interpret these models in a population context?

Ines: It means the way pLMs compute contacts is fundamentally different from how classical coevolutionary models compute them <ref:2611.16881#pg3>. The shared structure of the final contact score pipeline, whether it's the Potts tensor or the categorical Jacobian, is what makes this comparison possible <ref:2611.16881#pg3>.

Marcus: The paper shows that this gap between how these models compute contacts and how classical methods do is real. It also points out that increasing model size, going from eight million to three billion parameters, doesn't solve the problem; in fact, larger models are more responsive to biochemical classes but still degrade under relabeling <ref:2611.16881#pg2>.

Yuki: That tells us that scaling up the model isn't the answer for capturing true coevolutionary signals if the underlying mechanism is based on identity matching rather than pure statistical inference <ref:2611.16881#pg2>. So, what’s the implication for building predictive tools?

Ines: The implication is that we can use these diagnostic perturbations to expose hidden representation biases in these neural black boxes and stop over-relying on them without needing a full theory of coevolution <ref:2611.16881#pg3>. It’s about using the math itself as a probe for what the model is actually learning about the sequence structure <ref:2611.16881#pg3>.

Marcus: So to wrap up, this paper "Protein Language Models Fail To Capture Contact Predictions Under Invariant Relabeling" shows that high contact precision from protein language models doesn't automatically mean they are computing contacts through coevolution <ref:2611.16881#pg3>. It proves they rely on explicit residue identities or alphabet anchoring, which is a different mechanism than what classical methods use to separate direct coevolution from indirect correlation <ref:2611.16881#pg3>.

Yuki: And for us looking at the bigger picture, it means that when we trust these models for biological inference, we have to be careful about whether they're capturing deep evolutionary patterns or just surface-level sequence matching <ref:2611.16881#pg2>. That’s a big distinction in how you interpret the data coming out of these large language models.

Conclusion: Ines: So we're wrapping up this deep dive on these protein language models and their ability to predict contacts from a single sequence.

Marcus: Yeah, this paper, "Protein Language Models Fail To Capture Contact Predictions Under Invariant Relabeling," it basically shows that just because a model gets high accuracy on its predictions isn't proof that it's actually learning the deep evolutionary statistics we think it is.

Yuki: So if those models aren't capturing the coevolution, what are they really doing instead? Are they just memorizing patterns?

Ines: They seem to be relying on something more rigid, something like explicit residue identities or what the authors call "alphabet anchoring," rather than just looking at statistical relationships.

Marcus: That means when you swap out one amino acid for another within the same family, but cross a different biochemical class, the model breaks completely. It only works if you keep the letters in their original biochemical neighborhoods.

Yuki: So this suggests there’s a real difference between how these neural models are making predictions and how classical methods based on statistical inference actually work.

Ines: Exactly. The numbers show that when you test them with random relabeling, the accuracy drops from being very high to almost nothing, while classical estimators like the Potts model stay exactly the same because they're built around those column statistics.

Marcus: The authors point out that even as these models get bigger—we’re talking eight million to three billion parameters—they still degrade under this arbitrary relabeling test. Scaling up didn't fix their fundamental issue.

Yuki: That’s a pretty sobering thought for population genetics, meaning that just because we can train a massive model doesn't automatically mean it has captured the underlying biological truth about how proteins actually interact over evolutionary time.

Ines: It really highlights this gap between how these pLMs compute contacts and how classical coevolutionary models do, which opens up a new way to look at what these neural networks are actually learning.

Marcus: And this whole concept of "alphabet anchoring" could be the key to figuring out why we see such high precision in protein language models right now.

Paarth Batra, Aryan Keluskar

The Biodesign Institute

q-bio.QM, cs.AI

Submitted: 2026-11-12

Updated: 2026-11-12

Comments: Accepted to the ICBINB - Failure Modes of AI in Biology Workshop at NeurIPS 2026. 15 pages, 6 figures

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 89/100

The gist: Protein language models (pLMs) produce highly accurate contact maps from single-sequence inputs, and these maps collapse under arbitrary relabeling that leaves classical alignment-based methods…

Key concepts

Global Bijective Relabeling
This is a transformation where every unique amino acid letter in the protein sequence is swapped with another unique letter. Crucially, this process permutes the letters but leaves all alignment-level information, column entropies, and pairwise edit distances completely unchanged. It serves as a diagnostic tool to test if a model relies on specific residue identities.
Pseudolikelihood Potts Model
This is a classical statistical method used to predict protein contacts based on sequence alignments. The paper proves that this model is invariant under global relabeling because its objective function remains identical and the parameter space mapping is bijective. It serves as the benchmark for what contact prediction should look like if it relies only on general statistical properties.
Alphabet Anchoring
This phenomenon describes how pLMs fail when their alphabet is relabeled, even though classical models succeed. Instead of learning universal coevolutionary rules, the pLM implicitly learns to condition its predictions on the specific chemical identity of each residue. This 'anchoring' means the model relies on knowing which specific amino acid is at a position rather than just general evolutionary trends.
Column Statistics
These are summary measures derived from an alignment that describe the statistical properties of residues at specific positions across different sequences. The paper shows that estimators relying solely on these statistics, such as the inverse-covariance estimator, are provably invariant under relabeling because they only depend on the relative arrangement and frequency of residues, not their specific labels.

Summary

Protein language models (pLMs) produce highly accurate contact maps from single-sequence inputs, and these maps collapse under arbitrary relabeling that leaves classical alignment-based methods invariant The authors introduce a diagnostic framework using global bijective relabeling of the amino-acid alphabet to test whether pLM contact readouts rely on explicit residue identities rather than pure coevolutionary statistics This work demonstrates that while classical estimators like Potts models remain invariant under this transformation, pLMs fail the test because they rely on explicit, canonical residue identities or alphabet anchoring

Testing Invariance of Classical Estimators

The authors establish that any estimator reading only column statistics must be invariant under a global bijection because column entropies and the true contact map are unchanged by construction They prove this invariance for the pseudolikelihood Potts model and, in closed form, for the inverse-covariance estimator Specifically, Proposition 2 shows that the inverse-covariance estimator remains accurate under relabeling because it commutes with the permutation matrix transformation

Failure Modes of Protein Language Models

Every pLM tested degrades when its alphabet is globally relabeled, showing that they do not compute contacts solely from column statistics The failure is not tied to a single architecture or tokenizer, as the degradation replicates across ESM-2, MSA Transformer, and AMPLIFY Furthermore, the degradation costs accuracy; for ESM-2 650M on ubiquitin, precision drops from 2.33× the Potts precision to 0. p. 1

Where the paper says it

The failure also costs accuracy: on ubiquitin, over a candidate set where the two precisions are directly comparable, ESM-2 goes from 2. 33× the Potts precision to 0.

17× under relabeling

Distinguishing Identity from Edit Distance

The study isolates the cause of degradation by using three regimes—random, in-class, and cross-class permutations—to separate edit distance from biochemical class The authors introduce a matched-Hamming control to explicitly separate edit distance from residue identity by ensuring all arms mutate the same number of residues at the same positions This reveals that models rely on residue identity conditioned pattern matching rather than continuous distance metrics

Implications for Biological Modeling

The findings suggest a gap between how pLMs and classical coevolutionary models compute contacts, showing that high contact precision cannot be cited as proof that a model computes contacts through coevolution The phenomenon is termed alphabet anchoring, where the readout holds when residues are swapped within biochemical classes but collapses when swaps cross class boundaries This diagnostic perturbation exposes hidden representation biases and prevents over-reliance on neural black boxes in biological modeling The authors conclude that scaling does not close the gap, as larger models still degrade under arbitrary relabeling

Limitations of the Study

The scope is limited to a relatively small number of proteins and specific readouts, meaning the results are necessary but not sufficient evidence for coevolutionary inference The authors note that invariance under alphabet relabeling is a necessary condition for a readout to be a pure function of column statistics, but it is not sufficient to prove learned coevolution The study also did not test retrained or fine-tuned models, as the claim concerns public pre-trained checkpoints The work releases all code and data used for reproducibility

References

[2] Thomas M. Cover and Joy A. p. 7

Where the paper says it

[2] Thomas M. Cover and Joy A.

p. 7
Where the paper says it

[2] Thomas M. Cover and Joy A.

Thomas. Elements of Information Theory

[3] Margaret O. Dayhoff, Richard M. p. 7

Where the paper says it

[3] Margaret O. Dayhoff, Richard M.

p. 7
Where the paper says it

[3] Margaret O. Dayhoff, Richard M.

Schwartz, and Bruce C. p. 7
Where the paper says it

Dayhoff, Richard M. Schwartz, and Bruce C.

p. 7
Where the paper says it

Dayhoff, Richard M. Schwartz, and Bruce C.

Orcutt

[11] Cosmin Ieremie, Aled Ewing, and Mahesan Niranjan. Protein language models meet reduced amino acid alphabets

[12] David T. Jones, Daniel W. p. 8

Where the paper says it

[12] David T. Jones, Daniel W.

p. 8
Where the paper says it

[12] David T. Jones, Daniel W.

Buchan, Domenico Cozzetto, and Massimiliano Pontil. p. 8
Where the paper says it

Jones, Daniel W. Buchan, Domenico Cozzetto, and Massimiliano Pontil.

p. 8
Where the paper says it

Jones, Daniel W. Buchan, Domenico Cozzetto, and Massimiliano Pontil.

PSICOV: precise structural contact prediction using sparse inverse covariance estimation

[15] Debora S. Marks, Lucy J. p. 8

Where the paper says it

[15] Debora S. Marks, Lucy J.

p. 8
Where the paper says it

[15] Debora S. Marks, Lucy J.

Colwell, Robert Sheridan, Thomas A. p. 8
Where the paper says it

Marks, Lucy J. Colwell, Robert Sheridan, Thomas A.

p. 8
Where the paper says it

Marks, Lucy J. Colwell, Robert Sheridan, Thomas A.

Hopf, Andrea Pagnani, Riccardo Zecchina, and Chris Sander. p. 8
Where the paper says it

Colwell, Robert Sheridan, Thomas A. Hopf, Andrea Pagnani, Riccardo Zecchina, and Chris Sander.

p. 8
Where the paper says it

Colwell, Robert Sheridan, Thomas A. Hopf, Andrea Pagnani, Riccardo Zecchina, and Chris Sander.

Protein 3d structure computed from evolutionary sequence variation

[20] Faruck Morcos, Andrea Pagnani, Bryan Lunt, Annabella Bertolino, Debora S. Marks, Chris Sander, Riccardo Zecchina, José N. p. 8

Where the paper says it

[20] Faruck Morcos, Andrea Pagnani, Bryan Lunt, Annabella Bertolino, Debora S. Marks, Chris Sander, Riccardo Zecchina, José N.

p. 8
Where the paper says it

[20] Faruck Morcos, Andrea Pagnani, Bryan Lunt, Annabella Bertolino, Debora S. Marks, Chris Sander, Riccardo Zecchina, José N.

Onuchic, Terence Hwa, and Martin Weigt. p. 8
Where the paper says it

Marks, Chris Sander, Riccardo Zecchina, José N. Onuchic, Terence Hwa, and Martin Weigt.

p. 8
Where the paper says it

Marks, Chris Sander, Riccardo Zecchina, José N. Onuchic, Terence Hwa, and Martin Weigt.

Direct-coupling analysis of residue coevolution captures native contacts across many protein families

[23] Roshan M Rao, Jason Liu, Robert Verkuil, Joshua Meier, John Canny, Pieter Abbeel, Tom Sercu, and Alexander Rives. Msa transformer

[28] Jesse Vig, Ali Madani, Lav R Varshney, Caiming Xiong, Richard Socher, and Nazneen Fatema Rajani. Bertology meets biology: Interpreting attention in protein language models

[31] Zhidian Zhang, Hannah K Wayment-Steele, Garyk Brixi, Haobo Wang, Dorothee Kern, and Sergey Ovchinnikov. Protein language models learn evolutionary statistics of interacting sequence motifs

[26] Rome Thorstenson. Protein contacts are already in the attention: a single-forward-pass alternative to the categorical jacobian

[10] Steven Henikoff and Jorja G. Henikoff. Amino acid substitution matrices from protein blocks

[18] Jaina Mistry, Sara Chuguransky, Lowri Williams, Matloob Qureshi, Gustavo A Salazar, Erik LL Sonnhammer, Silvio CE Tosatto, Lisanna Paladin, Shriya Raj, Lorna J Richardson

[19] Bohdan Monastyrskyy, Daniel d’Andrea, Krzysztof Fidelis, Anna Tramontano, and Andriy Kryshtafovych. Evaluation of residue–residue contact prediction in casp10

[21] Lynne Reed Murphy, Anders Wallqvist, and Ronald M Levy. Simplified amino acid alphabets for protein fold recognition and implications for folding

[9] Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A. Wichmann. Shortcut learning in deep neural networks

[13] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models

[24] Alexander Rives, Joshua Meier, Tom Sercu, Siddharth Goyal, Zeming Lin, Jason Liu, Demi Guo, Myle Ott, C. Lawrence Zitnick, Jerry Ma

[25] Stefan Seemayer, Michael Gruber, and Johannes Söding. CCMpred—fast and precise prediction of protein residue–residue contacts

--- Page 1 ---

Protein language models (pLMs) produce highly accurate contact maps from single-sequence inputs, and these maps collapse under arbitrary relabeling that leaves classical alignment-based methods invariant This work demonstrates that while classical estimators like Potts models remain invariant under this transformation, pLMs fail the test because they rely on explicit, canonical residue identities or alphabet anchoring

[1] Helen M Berman, Tammy Battistuz, Talapady N Bhat, Wolfgang F Bluhm, Philip E Bourne, Kyle Burkhardt, Zukang Feng, Gary L Gilliland, Lisa Iype, Shri Jain

[2] Thomas M. Cover and Joy A. Thomas. Elements of Information Theory

[3] Margaret O. Dayhoff, Richard M. Schwartz, and Bruce C. Orcutt

[11] Cosmin Ieremie, Aled Ewing, and Mahesan Niranjan. Protein language models meet reduced amino acid alphabets

[12] David T. Jones, Daniel W. Buchan, Domenico Cozzetto, and Massimiliano Pontil. PSICOV: precise structural contact prediction using sparse inverse covariance estimation

[15] Debora S. Marks, Lucy J. Colwell, Robert Sheridan, Thomas A. Hopf, Andrea Pagnani, Riccardo Zecchina, and Chris Sander. Protein 3d structure computed from evolutionary sequence variation

[20] Faruck Morcos, Andrea Pagnani, Bryan Lunt, Annabella Bertolino, Debora S. Marks, Chris Sander, Riccardo Zecchina, José N. Onuchic, Terence Hwa, and Martin Weigt. Direct-coupling analysis of residue coevolution captures native contacts across many protein families

[23] Roshan M Rao, Jason Liu, Robert Verkuil, Joshua Meier, John Canny, Pieter Abbeel, Tom Sercu, and Alexander Rives. Msa transformer

[28] Jesse Vig, Ali Madani, Lav R Varshney, Caiming Xiong, Richard Socher, and Nazneen Fatema Rajani. Bertology meets biology: Interpreting attention in protein language models

[31] Zhidian Zhang, Hannah K Wayment-Steele, Garyk Brixi, Haobo Wang, Dorothee Kern, and Sergey Ovchinnikov. Protein language models learn evolutionary statistics of interacting sequence motifs

[26] Rome Thorstenson. Protein contacts are already in the attention: a single-forward-pass alternative to the categorical jacobian

[10] Steven Henikoff and Jorja G. Henikoff. Amino acid substitution matrices from protein blocks

[18] Jaina Mistry, Sara Chuguransky, Lowri Williams, Matloob Qureshi, Gustavo A Salazar, Erik LL Sonnhammer, Silvio CE Tosatto, Lisanna Paladin, Shriya Raj, Lorna J Richardson

[19] Bohdan Monastyrskyy, Daniel d’Andrea, Krzysztof Fidelis, Anna Tramontano, and Andriy Kryshtafovych. Evaluation of residue–residue contact prediction in casp10

[21] Lynne Reed Murphy, Anders Wallqvist, and Ronald M Levy. Simplified amino acid alphabets for protein fold recognition and implications for folding

[9] Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A. Wichmann. Shortcut learning in deep neural networks

[13] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models

[24] Alexander Rives, Joshua Meier, Tom Sercu, Siddharth Goyal, Zeming Lin, Jason Liu, Demi Guo, Myle Ott, C. Lawrence Zitnick, Jerry Ma

[25] Stefan Seemayer, Michael Gruber, and Johannes Söding. CCMpred—fast and precise prediction of protein residue–residue contacts

--- Page 11 ---

Shared contact pipeline. Potts models and pLMs both yield a coupling tensor of shape L×q×L×q: Potts from its fitted couplings Jij or a regularized precision matrix, and a pLM from its categorical Jacobian J[i, a, j, b], the change in the logit for residue b at position j when position i is set to residue a [31] The same pipeline turns either into an L × L contact score S: mean-centering over both amino-acid axes, symmetrization, the Frobenius norm of each q × q block, and average product correction (APC) [6] For Potts, every input to this pipeline is a function of the alignment’s column statistics, which is what makes the symmetry below available

Invariance Proofs

Proposition 1 proves that pseudolikelihood Potts models are invariant under global relabeling because the objective function remains identical and parameter space maps are bijective Proposition 2 proves that the inverse-covariance estimator is exactly invariant because the transformation preserves Euclidean norms and commutes with all scoring steps

Experimental Setup Details

The authors detail the experimental setup, including using nine proteins for ESM-2 650M sweep and contact-head replication They specify the relabeling regimes, noting that at nominal m=20, in-class moves 15 residue types and cross-class moves 14 to 16 The matched-Hamming control is constructed to ensure all three arms mutate the same number of residues at the same positions

Metrics and Results Summary

The primary metrics reported are Overlap, defined as the fraction of a model’s top-L/2 long-range contacts on the original sequence that it also predicts on the relabeled one, and Precision P@L/2 measured against solved structures Results show that ESM-2 650M's overlap under random relabeling falls to 0.015 at m = 20, while in-class relabeling keeps 0. p. 3

Where the paper says it

Averaged over nine proteins and three seeds, ESM-2 650M’s overlap under random relabeling falls to 0. 015 at m = 20, while in-class relabeling keeps 0.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Alphabet anchoring detection for pLMs: Implement a diagnostic framework using global bijective relabeling of amino acids to test if contact maps rely on explicit residue identity rather than pure coevolutionary statistics. This allows researchers to distinguish between models that use alphabet anchoring and those that compute contacts from column statistics alone, as proven by the observation that A model whose contact predictions change under relabeling is not computing them from column statistics alone.

  2. Biochemical class-aware feature weighting: Modify pLM training or inference to incorporate knowledge of biochemical similarity classes during prediction, allowing the system to maintain performance when residues are swapped within groups and degrade predictably only when swaps cross class boundaries. This leverages the finding that The readout holds when residues are swapped within biochemical classes and collapses when swaps cross class boundaries.

  3. Scale-aware regularization: Adjust model scaling strategies to recognize that larger models become more sensitive to biochemical similarity, while arbitrary relabeling remains destructive across all sizes. This prevents over-reliance on high precision scores, as the paper shows high precision cannot be cited as proof that a pLM computes contacts through coevolution.

Abstract

Protein language models (pLMs) produce highly accurate contact maps from single-sequence inputs. People often take this accuracy as proof that these models implicitly learn invariant coevolutionary statistics, extracting the same signals as classical alignment-based methods like Potts. In this work, we introduce a diagnostic framework around a global bijective relabeling of the amino-acid alphabet. This transformation permutes residue letters while leaving all alignment-level information, column entropies, and pairwise edit distances unchanged. Pseudolikelihood Potts models and inverse-covariance estimators remain invariant under this transformation, but pLMs fail the test. Across multiple architectures (ESM-2, MSA Transformer, AMPLIFY) and model scales (8M to 3B parameters), pLM contact maps collapse under arbitrary relabeling. Using Hamming distance controls and biochemical grouping interventions, we attempt to isolate what likely causes this degradation and find that these models rely on explicit, canonical residue identities (``alphabet anchoring'') rather than pure coevolutionary statistics. Increasing scale makes models more responsive to biochemical classes, but their performance still degrades under relabeling. Our findings show a gap between how pLMs and classical coevolutionary models compute contacts. More broadly, our work shows how diagnostic perturbations that leave classical models provably unchanged can expose hidden representation biases and prevent over-reliance on neural black boxes in biological modeling.

Sources

Related papers