C3P: Contrastive promoter-protein pretraining yields representations capturing bacterial gene regulation
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Genomics Radio. Generated commentary on the latest computational biology and genomics papers.
Ines: I'm Ines.
Marcus: And I'm Marcus, here with Yuki.
Ines: We're looking at the paper: "C3P: Contrastive promoter-protein pretraining yields representations capturing bacterial gene regulation".
Marcus: Here's the gist of the paper. Despite the increasing scale of genome language models (gLMs), their ability to decode the function of regulatory sequences remains unclear<ref:2605.25242#pg2> Self-supervised contrastive approaches provide a promising alternative<ref:2605.25242#pg2>.
Ines: By learning to align promoters to their corresponding proteins, we leverage the rich representations of proteins learned by protein language models as supervisory signal for the learning of promoter representations<ref:2605.25242#pg4>.
Marcus: After training on 88 million bacterial promoter-protein pairs, we evaluate the predictive power of C3P-learned promoter representations for inference of curated regulatory annotations, finding multi-fold improvement over leading gLMs<ref:2605.25242#pg2>.
Ines: We also introduce zero-shot co-regulated gene retrieval, the ability to find co-regulated genes in a genome using no experimental data<ref:2605.25242#pg3>.
Paper summary: Ines: So we've covered the basic setup. To expand on what C3P actually does, it’s inspired by how language-image models like CLIP work. They are learning to align two different types of sequences: DNA promoter sequences and the protein sequences they control >
Marcus: The methodology involves tokenizing the promoter sequence into overlapping three-mers and encoding it with a promoter encoder, while using a pre-trained protein encoder, specifically ESM2, for the protein sequence >
Yuki: So you have these two encoders generating embeddings, which are then passed through learnable linear projection layers so they can share a common space where alignment happens >
Ines: The training uses symmetric InfoNCE loss. This loss function makes the model maximize the cosine similarity between correct promoter-protein pairs while minimizing similarity between incorrect pairings across a large batch of data >
Marcus: They trained this on eighty-eight million bacterial promoter-protein pairs and then tested its predictive power on inferring curated regulatory annotations for a held-out species > <ref:2605.25242#pg1,on 88 million bacterial promoter-protein pairs>
Yuki: And according to the paper, they found that these C3P learned representations showed multi-fold improvement over leading gLMs when predicting those annotations >
Ines: They also showed that this approach provides significant zero-shot performance gains compared to a randomly initialized baseline, unlike what they saw with the standard gLMs >
Marcus: The key thing here is the scaling analysis. They managed to achieve strong performance at a fraction of the training cost required by leading gLMs >
Yuki: So, while it’s better than existing methods on this specific task, you have to consider that they are still comparing this against models that already learn about protein structure well >
Ines: Right. The paper shows that the promoter embeddings learned by C3P actually outperformed the ESM2 protein embeddings on their own when tested >
Conclusion: Marcus: So wrapping up the "C3P: Contrastive promoter-protein pretraining yields representations capturing bacterial gene regulation" paper, it seems like this self-supervised contrastive alignment method is a promising path for learning about regulatory sequences in genomes >
Ines: It moves away from the struggle of standard language models by using protein knowledge as a guide to learn what promoters actually do >
Yuki: For the wider picture, if this method can indeed infer regulation from DNA alone, it opens up the possibility of building complete models of cells entirely from their genomic sequence, which is a huge concept for understanding life >
Marcus: That zero-shot co-regulated gene retrieval capability is what really stands out as something that could change how we approach regulatory genomics across different bacterial datasets >
Ines: It’s important to remember the paper's caveat. They noted that while this method improves prediction, it still doesn't fully decode the function of regulatory sequences in a definitive way yet >
Yuki: So, it’s an improvement over what we have now, but it’s not the final answer for understanding all bacterial regulation >
Marcus: Exactly. The success is in showing that this contrastive alignment approach works well with just sequence data and existing protein knowledge to recover regulatory signals better than current gLMs >
University of Toronto
q-bio.GN
Submitted: 2026-05-24
Updated: 2026-11-05
Comments: Accepted (Oral) at 40th Conference on Neural Information Processing Systems (NeurIPS 2026)
Code: https://github.com/dufaultc/contrastive-promoter-protein-pretraining
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 96/100
The gist: Despite the increasing scale of genome language models (gLMs), their ability to decode the function of regulatory sequences remains unclear<ref:2605.25242#pg2> Self-supervised contrastive approaches…
Key concepts
- Genome Language Models (gLMs)
- These are deep learning models trained on DNA sequences to learn general representations of genomes. The paper investigates whether these models naturally capture the function of regulatory sequences, finding they often struggle in this specific area.
- Protein Language Models (pLMs)
- These models analyze amino acid sequences from proteins to understand their structure and function. Their rich protein representations are used as a supervisory signal to train the promoter encoder in the C3P approach.
- Contrastive Learning (InfoNCE Loss)
- This training technique forces the model to learn an alignment between two different types of data—promoters and proteins. The model is trained to maximize similarity between correct pairs while minimizing similarity between incorrect pairings, effectively teaching it what a functional promoter sequence looks like.
- Zero-shot Co-regulated Gene Retrieval
- This is the ability to find genes that are regulated together in a genome without any prior experimental data for those specific genes. C3P training significantly improves this capability compared to standard gLMs.
Terminology
Summary
Despite the increasing scale of genome language models (gLMs), their ability to decode the function of regulatory sequences remains unclear<ref:2605. p. 0 C3P: Contrastive promoter-protein pretraining yields
representations capturing bacterial gene regulation
Cameron Dufault∗
Department of Computer Science
University of Toronto
Scott Xu
Department of Computer Science
University of Toronto
Alan M. Moses
Department of Cell and Systems Biology
Department of Computer Science
University of Toronto
Abstract
Despite the increasing scale of genome language models (gLMs), their ability to
decode the function of regulatory sequences remains unclear. gLM pretraining
relies on sequence reconstruction, which may struggle due to the noisy, rapidly
evolving nature of regulatory DNA. Self-supervised contrastive approaches provide
a promising alternative. Inspired by language-image architectures like CLIP, we
introduce contrastive promoter-protein pretraining (C3P). By learning to align
promoters to their corresponding proteins, we leverage the rich representations of
proteins learned by protein language models as supervisory signal for the learning
of promoter representations. By learning to align
promoters to their corresponding proteins, we leverage the rich representations of
proteins learned by protein language models as supervisory signal for the learning
of promoter representations. After training on 88 million bacterial promoter-protein
pairs, we evaluate the predictive power of C3P-learned promoter representations
for inference of curated regulatory annotations, finding multi-fold improvement
over leading gLMs. After training on 88 million bacterial promoter-protein
pairs, we evaluate the predictive power of C3P-learned promoter representations
for inference of curated regulatory annotations, finding multi-fold improvement
over leading gLMs. We also introduce zero-shot co-regulated gene retrieval, the
ability to find co-regulated genes in a genome using no experimental data. We also introduce zero-shot co-regulated gene retrieval, the
ability to find co-regulated genes in a genome using no experimental data. We
find that compared to a randomly initialized baseline, C3P training consistently
provides significant zero-shot performance gains, unlike gLMs. We also introduce zero-shot co-regulated gene retrieval, the
ability to find co-regulated genes in a genome using no experimental data. We
find that compared to a randomly initialized baseline, C3P training consistently
provides significant zero-shot performance gains, unlike gLMs.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Introduction and Motivation
The biological effects of genes are determined by both the gene product (protein, tRNA, etc.) and when and where the product is expressed<ref:2605. p. 0 1 Introduction
The biological effects of genes are determined by both the gene product (protein, tRNA, etc. ) and
when and where the product is expressed. ) and
when and where the product is expressed. Deep learning has revolutionized the tools for computational
prediction of structure and function of proteins directly from their amino acid sequences [15], but
predicting the regulation of gene expression directly from DNA sequences remains a long-standing
challenge. ) and
when and where the product is expressed. Deep learning has revolutionized the tools for computational
prediction of structure and function of proteins directly from their amino acid sequences [15], but
predicting the regulation of gene expression directly from DNA sequences remains a long-standing
challenge. Without regulatory context, we cannot determine from the genome sequence alone how
the expression of encoded genes is orchestrated to build the cell and adapt to the environment [34]. Supervised deep learning approaches have emerged for inferring regulation from DNA [2, 3, 21],
but require training on experimental data which is unavailable for the vast majority of organisms,
∗dufaultc@cs. Despite the relative simplicity of their genomes,
where the regulatory sequences are primarily contained in the regions between proteins (hereafter
referred to as promoters), for most bacterial species we know little about their gene regulation [34, 4]. With the vast quantity of unlabeled DNA sequencing data now available, there is clear opportunity
for unsupervised approaches to help us learn regulatory function. With the vast quantity of unlabeled DNA sequencing data now available, there is clear opportunity
for unsupervised approaches to help us learn regulatory function. Existing unsupervised approaches
to learning from the genome have centered on language models [7]. Existing unsupervised approaches
to learning from the genome have centered on language models [7]. However, while protein language
models (pLMs), which operate on sequences of amino acids, succeed in learning representations
capturing the structure and function of proteins [20, 37], multiple analyses have called into question
whether existing genome language models (gLMs), which operate on DNA sequences, learn about
gene regulation from their pretraining [45, 49]. However, while protein language
models (pLMs), which operate on sequences of amino acids, succeed in learning representations
capturing the structure and function of proteins [20, 37], multiple analyses have called into question
whether existing genome language models (gLMs), which operate on DNA sequences, learn about
gene regulation from their pretraining [45, 49]. Specific demonstrations that such models learn to
capture bacterial regulatory sequence function have been limited.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
How the training signal is built
The C3P approach is inspired by language-image architectures like CLIP [35]<ref:2605.25242#pg3>. The objective is to align the learned representations of promoters with the representations of their corresponding proteins<ref:2605. p. 3 1 Objective and Architecture
An overview of the C3P architecture and training task is shown in Figure 1. Inspired by languageimage models such as CLIP [35], the objective is to align the learned representations of promoters
with the representations of their corresponding proteins. Inspired by languageimage models such as CLIP [35], the objective is to align the learned representations of promoters
with the representations of their corresponding proteins. Given a promoter-protein pair, the promoter sequence is tokenized into overlapping 3-mers and
encoded by the promoter encoder, a randomly initialized transformer encoder [48]. Given a promoter-protein pair, the promoter sequence is tokenized into overlapping 3-mers and
encoded by the promoter encoder, a randomly initialized transformer encoder [48]. The protein
sequence is encoded by a frozen pretrained protein encoder, for which we utilize ESM2 (150M) [20],
with embeddings averaged over the sequence length. The protein
sequence is encoded by a frozen pretrained protein encoder, for which we utilize ESM2 (150M) [20],
with embeddings averaged over the sequence length. The outputs of both encoders are then passed
through learnable linear projection layers to have shared dimensionality (dprojection = 256). The outputs of both encoders are then passed
through learnable linear projection layers to have shared dimensionality (dprojection = 256). The model is trained with symmetric InfoNCE loss [30]. Given a batch of N promoter-protein pairs,
let z
prom
i
and z
prot
i
denote the L2-normalized, projected representations of the i-th promoter and
protein, respectively. For each batch, the model learns to maximize the cosine similarity between
the N correct promoter-protein pairs while minimizing the similarity between the N2 − N incorrect
pairings.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the gains come from
The multi-fold improvement over leading gLMs is demonstrated through several evaluations<ref:2605.25242#pg2>.
-
Nearest neighbour prediction of curated regulatory annotations in a held-out species shows multi-fold improvement over leading gLMs at capturing regulatory function in bacterial promoter representations, despite having orders of magnitude fewer parameters<ref:2605. p. 1
25242#pg2>.Where the paper says it
After training on 88 million promoter-protein pairs from ∼23,000 bacterial species, C3P models show a multi-fold improvement over leading gLMs at capturing regulatory function in bacterial promoter representations, despite having orders of magnitude fewer parameters. We evaluate this through nearest neighbour prediction of curated regulatory annotations in a held-out species, as well as performance at zero-shot co-regulated gene retrieval, an evaluation framework we introduce for exploring the potential for solving a grand challenge of regulatory genomics: inferring regulation from the genome alone.
-
Zero-shot co-regulated gene retrieval, an evaluation framework we introduce for exploring the potential for solving a grand challenge of regulatory genomics: inferring regulation from the genome alone<ref:2605. p. 1
25242#pg2>. We find that compared to a randomly initialized baseline, C3P training consistently provides significant zero-shot performance gains, unlike gLMs<ref:2605.25242#pg2>.Where the paper says it
After training on 88 million promoter-protein pairs from ∼23,000 bacterial species, C3P models show a multi-fold improvement over leading gLMs at capturing regulatory function in bacterial promoter representations, despite having orders of magnitude fewer parameters. We evaluate this through nearest neighbour prediction of curated regulatory annotations in a held-out species, as well as performance at zero-shot co-regulated gene retrieval, an evaluation framework we introduce for exploring the potential for solving a grand challenge of regulatory genomics: inferring regulation from the genome alone.
-
C3P models show substantial improvement over their random baselines at top-1 full and any match accuracy on all 25 iModulonDB datasets<ref:2605. p. 7
25242#pg7>.Where the paper says it
C3P demonstrates multi-fold improvement over its random baselines across nearly all datasets, while other approaches broadly show minor improvement or in some cases worsened performance (Figure 5, Appendix F). 4 C3P demonstrates favorable scaling for regulatory inference compared to gLMs Noting that Evo2 showed evidence of improvement in only 3 (top-1 full match) and 2 (top-1 any match) more datasets than PromoGen2, despite having ∼45x more parameters, we next performed an analysis of the scaling of C3P and compared with the empirical trend of our gLM baselines.
Scaling and Generalization
The scaling analysis reveals the potential for further improvement as well as the efficiency of C3P, which achieved strong performance at a fraction of the training cost of leading gLMs<ref:2605.25242#pg2>. The success of C3P at zero-shot co-regulated gene retrieval suggests the possibility of inferring the regulatory networks of entirely unstudied species, an important step towards the ability to build complete models of cells entirely from the genome [34]<ref:2605. p. 9 Furthermore,
we confirmed that C3P consistently leads to significant performance improvement over a randomly
initialized baseline, unlike gLMs. The success of C3P at zero-shot co-regulated gene retrieval suggests
the possibility of inferring the regulatory networks of entirely unstudied species, an important step
towards the ability to build complete models of cells entirely from the genome [34]. Furthermore,
we confirmed that C3P consistently leads to significant performance improvement over a randomly
initialized baseline, unlike gLMs. The success of C3P at zero-shot co-regulated gene retrieval suggests
the possibility of inferring the regulatory networks of entirely unstudied species, an important step
towards the ability to build complete models of cells entirely from the genome [34]. We believe that this is due to regulatory sequences being better suited
for joint-embedding based self-supervised learning objectives than reconstruction-based objectives
[1], as well as our use of a pretrained pLM. C3P promoter embeddings also outperformed ESM2
(150M) protein embeddings.Where the paper says it
Where the paper says it
Where the paper says it
Conclusion
Genome language models are known to struggle to decode the function of regulatory sequences[45, 49]. Inspired by the success of multi-modal contrastive alignment in the language-image domain [35], we introduced C3P, a novel self-supervised approach for learning representations of regulatory sequences<ref:2605. p. 9 6 Conclusion
Genome language models are known to struggle to decode the function of regulatory sequences
[45, 49]. Inspired by the success of multi-modal contrastive alignment in the language-image domain
[35], we introduced C3P, a novel self-supervised approach for learning representations of regulatory
sequences. Inspired by the success of multi-modal contrastive alignment in the language-image domain
[35], we introduced C3P, a novel self-supervised approach for learning representations of regulatory
sequences. Using C3P, we trained a promoter encoder on 88 million bacterial promoter-protein
pairs and evaluated its performance on nearest-neighbour prediction of gold-standard regulatory
annotations from a held-out species, as well as zero-shot co-regulated gene retrieval across 25 diverse
bacterial datasets. Using C3P, we trained a promoter encoder on 88 million bacterial promoter-protein
pairs and evaluated its performance on nearest-neighbour prediction of gold-standard regulatory
annotations from a held-out species, as well as zero-shot co-regulated gene retrieval across 25 diverse
bacterial datasets. We demonstrated that C3P representations are substantially more predictive of
gene regulation than leading gLMs, despite requiring orders of magnitude less compute. We demonstrated that C3P representations are substantially more predictive of
gene regulation than leading gLMs, despite requiring orders of magnitude less compute. Furthermore,
we confirmed that C3P consistently leads to significant performance improvement over a randomly
initialized baseline, unlike gLMs.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Acknowledgments
We thank David Knowles, Alex Lu, Philip Fradkin, Andrew Duncan, Ami Sangster, and Rain Jin for feedback on the manuscript<ref:2605.25242#pg2>. CD and AMM acknowledge support from the Canada Research Chairs program<ref:2605. p. 9 7 Acknowledgments
We thank David Knowles, Alex Lu, Philip Fradkin, Andrew Duncan, Ami Sangster, and Rain Jin
for feedback on the manuscript. CD and AMM acknowledge support from the Canada Research
Chairs program. CD and AMM acknowledge support from the Canada Research
Chairs program. Computing resources used in preparing this research were provided, in part, by
the Province of Ontario, the Government of Canada through CIFAR, and companies sponsoring the
Vector Institute.Where the paper says it
Where the paper says it
Data and Code Availability
All data used in this work was sourced from publicly available databases<ref:2605.25242#pg2>. Codes for downloading data, extracting promoter protein pairs, and training C3P models are available at https://github. p. 9 8 Data and Code Availability
All data used in this work was sourced from publicly available databases. Codes for downloading data, extracting promoter protein pairs, and training C3P models are available at
https://github. Codes for downloading data, extracting promoter protein pairs, and training C3P models are available at
https://github. com/dufaultc/contrastive-promoter-protein-pretraining. com/dufaultc/contrastive-promoter-protein-pretraining. Trained models available at
https://huggingface. Codes for downloading data, extracting promoter protein pairs, and training C3P models are available at
https://github. com/dufaultc/contrastive-promoter-protein-pretraining.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Contrastive promoter-protein pretraining (C3P) can be used to learn representations of bacterial regulatory sequences by
learning to align promoters to their corresponding proteins,
which allows for capturing functional protein features in promoter representations, assimilar protein embeddings signals similar function.
-
The C3P model can perform zero-shot co-regulated gene retrieval, which is the
ability to find co-regulated genes in a genome using no experimental data,
suggesting the possibility ofinferring the regulatory networks of entirely unstudied species.
-
C3P models show a multi-fold improvement over leading genome language models at capturing regulatory function, and they
consistently provide significant improvement over randomly initialized baselines at zero-shot co-regulated gene retrieval, unlike gLMs.
-
The learned promoter representations from C3P can be used for nearest neighbour prediction of curated regulatory annotations with a multi-fold improvement over leading gLMs, demonstrating the
predictive power of C3P-learned promoter representations for inference of curated regulatory annotations.
Abstract
Despite the increasing scale of genome language models (gLMs), their ability to decode the function of regulatory sequences remains unclear. gLM pretraining relies on sequence reconstruction, which may struggle due to the noisy, rapidly evolving nature of regulatory DNA. Self-supervised contrastive approaches provide a promising alternative. Inspired by language-image architectures like CLIP, we introduce contrastive promoter-protein pretraining (C3P). By learning to align promoters to their corresponding proteins, we leverage the rich representations of proteins learned by protein language models as supervisory signal for the learning of promoter representations. After training on 88 million bacterial promoter-protein pairs, we evaluate the predictive power of C3P-learned promoter representations for inference of curated regulatory annotations, finding multi-fold improvement over leading gLMs. We also introduce zero-shot co-regulated gene retrieval, the ability to find co-regulated genes in a genome using no experimental data. We find that compared to a randomly initialized baseline, C3P training consistently provides significant zero-shot performance gains, unlike gLMs. Scaling analysis reveals the potential for further improvement as well as the efficiency of C3P, which achieved strong performance at a fraction of the training cost of leading gLMs. In addition to demonstrating that C3P training is effective for learning representations of bacterial regulatory sequences, our strong zero-shot co-regulated gene retrieval performance suggests the possibility of decoding gene regulation for millions of bacteria from their genomes alone.
Sources
- Exploring Simple Siamese Representation Learning
- Scaling Laws for Neural Language Models
- Evolution Is All You Need: Phylogenetic Augmentation for Contrastive Learning
- UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction
- Representation Learning with Contrastive Predictive Coding
- RoFormer: Enhanced Transformer with Rotary Position Embedding
Related papers
- SeekRBP: Leveraging Sequence-Structure Integration with Reinforcement Learning for Receptor-Binding Protein Identification
- Genome-Factory: A Library for Tuning, Deploying, and Interpreting Genomic Foundation Models
- AVA: Agentic Variant Adjudication for Automated, Auditable Read-Level Review in Cancer Diagnostic Sequencing
- Leveraging genomic deep learning models for the prediction of non-coding variant effects
- Adaptive Allele-Haplotype Incidence Indexing for Pangenome Queries: Representation, Reconstruction, and Benchmarking
- Bridging Graph and Pairwise Learning for Gene Regulatory Network Inference from scRNA-seq Data