UltraBERT: Beyond the Limit of BERT Pre-Training
cs.CL, cs.AI
Submitted: 2026-10-10
Updated: 2026-10-10
Code: https://github.com/allenai/pes2o
Sources
- LFM2 Technical Report
- Program Synthesis with Large Language Models
- Fine-Tuned 'Small' LLMs (Still) Significantly Outperform Zero-Shot Generative AI Models in Text Classification
- Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
- Training Verifiers to Solve Math Word Problems
- Efficient pre-training objectives for Transformers
- Gemma 2: Improving Open Language Models at a Practical Size
- Gemma 3 Technical Report
- CodeSearchNet Challenge: Evaluating the State of Semantic Code Search
- JetParticle-JEPA: An Efficient Self-Supervised Representation Learning method for Jet Tagging in High-Energy Physics
- Towards General Text Embeddings with Multi-stage Contrastive Learning
- Ministral 3
- Skywork-Reward: Bag of Tricks for Reward Modeling in LLMs
- RoBERTa: A Robustly Optimized BERT Pretraining Approach
- mmBERT: A Modern Multilingual Encoder with Annealed Language Learning
- MS MARCO: A Human Generated MAchine Reading COmprehension Dataset
- 2 OLMo 2 Furious
- From Topology to Retrieval: Decoding Embedding Spaces with Unified Signatures
- DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter
- GLU Variants Improve Transformer
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering