Multi-modal transformer for signal classification in nanopore blockade experiments

The paper introduces a multi-modal transformer architecture for classifying peptides and proteins from nanopore blockade experiments. The model jointly processes three representations of the same ionic current signal: raw time-series data, wavelet-transformed images, and static feature vectors (catch22 descriptors). The architecture extends the Vision Transformer framework, with separate branches for each modality and cross-attention mechanisms to fuse information between branches.

On a 42-peptide benchmark dataset containing 350,000 events, the multi-modal model achieves macro-averaged accuracy of 92.6% and micro-averaged accuracy of 92.1%, surpassing the best single-modality baseline (ResNet18 on wavelets, 81.7% macro / 81.5% micro) by more than 10 percentage points. The worst-performing class accuracy improves from 58.7% (best baseline) to 77.2%, and the best-performing class reaches 100% accuracy. Per-class accuracy distributions show the multi-modal model concentrates sharply above 0.75, with more than 15 of 42 classes at near-perfect accuracy, whereas single-modality models show broader distributions with tails of poorly classified classes.

The paper demonstrates that each modality captures complementary information: the top-10 classes for each individual modality differ, and the multi-modal model surpasses the performance of individual models on all three sets of top-performing classes. Attention analysis using attention rollout reveals that the time-series model attends to entry and exit phases of events and to deep, short blockades in the center of the signal, while the wavelet model attends to entry and exit phases and to the high-frequency band throughout the event. This indicates the two representations expose different aspects of the same event.

Transfer learning experiments on the XR7 dataset (20 amino acids linked to arginine heptapeptide carriers, 38,000 events) show that a model pre-trained on the peptide-ladder dataset and fine-tuned on XR7 converges faster, reaches slightly higher final accuracy, and exhibits a markedly narrower per-class accuracy distribution (worst class exceeding 90%) compared to a model trained from scratch (worst class as low as 70%). The transferred model achieves 99.0% macro-average accuracy on XR7, whereas previous work reported only 13 of 20 amino acids distinguishable by mean residual current alone.

The paper estimates 3–5% label noise in the peptide-ladder dataset, suggesting the achieved performance approaches the ceiling attainable on that dataset. The architecture places no restriction on the number or type of input branches, and could be extended to additional measurements such as electro-optical readouts. Limitations include that both datasets were recorded with the same aerolysin pore (demonstrating transfer across analytes but not across pore architectures) and that 42 peptide classes remain a small fraction of clinical diversity.

Improvements to AI systems:

1. Multi-modal fusion via cross-attention for heterogeneous signal representations – The architecture jointly processes raw time-series, wavelet-transformed images, and static feature vectors (catch22) with separate transformer branches and cross-attention. An improved AI system can dynamically weight complementary information from different views of the same signal, reducing reliance on any single feature extraction method and improving robustness to noise or missing modalities.

2. Attention rollout for interpretable signal segmentation – By analyzing attention patterns, the system can identify which temporal regions (e.g., entry/exit phases, deep short blockades) and frequency bands are most discriminative. An improved AI system can automatically highlight biologically relevant sub-events within a signal, enabling researchers to focus on critical regions for downstream analysis or experimental design.

3. Transfer learning with class-distribution-aware fine-tuning – Pre-training on a large, diverse peptide dataset and fine-tuning on a smaller target dataset (XR7) yields faster convergence, higher final accuracy (99.0% macro), and dramatically narrower per-class accuracy (worst class >90% vs. 70% from scratch). An improved AI system can generalize across different analyte sets with minimal labeled data, reducing the need for large annotated datasets in new nanopore applications.

4. Per-class performance monitoring and ceiling estimation – The system quantifies label noise (3–5%) and tracks worst-class accuracy, allowing it to estimate the theoretical performance ceiling. An improved AI system can flag when further accuracy gains are unlikely due to data quality, guiding users to invest in cleaner labels or more diverse training data rather than model tweaks.

5. Modality-agnostic extensible architecture – The framework supports any number of input branches (e.g., electro-optical readouts). An improved AI system can seamlessly integrate new sensor modalities without redesigning the core model, enabling multi-sensor fusion for richer biological characterization (e.g., combining ionic current with fluorescence or impedance data).

6. Class-specific attention-based feature prioritization – Since top-10 classes differ per modality, the system can learn which modality is most informative for each class. An improved AI system can adaptively select or weight branches per prediction, improving accuracy on rare or confusable classes (e.g., worst class improved from 58.7% to 77.2%).

7. Transfer across analytes with shared pore architecture – The system demonstrates that features learned on one analyte set (peptides) transfer to another (amino acid conjugates) using the same pore. An improved AI system can be pre-trained on a broad pore-specific corpus and fine-tuned for new analytes, accelerating deployment in clinical or environmental sensing without retraining from scratch.

What the improved AI system can do:

• Classify peptides/proteins from nanopore signals with >92% accuracy, near the theoretical limit given label noise.

• Identify which parts of a signal (e.g., blockade depth, duration, frequency content) drive classification, aiding mechanistic understanding.

• Rapidly adapt to new analyte panels with minimal labeled data, achieving >99% accuracy on unseen amino acid conjugates.

• Provide per-class confidence and flag low-confidence predictions for manual review, reducing errors in high-stakes applications.

• Integrate future sensor modalities (e.g., optical, electrochemical) without architectural changes, enabling multi-parameter single-molecule analysis.

• Estimate data quality and predict when additional training data will not improve performance, saving computational resources.