Nemotron 3 Ultra: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning
cs.CL, cs.AI, cs.LG
Submitted: 2026-06-12
Updated: 2026-06-12
Code: https://github.com/NVIDIA-NeMo/Nemotron
License: http://creativecommons.org/licenses/by/4.0/
The gist: We introduce Nemotron 3 Ultra, a 550 billion total and 55 billion active parameter Mixture-of-Experts Hybrid Mamba-Attention language model.
Terminology
Abstract
We introduce Nemotron 3 Ultra, a 550 billion total and 55 billion active parameter Mixture-of-Experts Hybrid Mamba-Attention language model. We pre-trained Nemotron 3 Ultra on 20 trillion text tokens, then extended the context length to 1M tokens, and post-trained using Supervised Fine Tuning (SFT), Reinforcement Learning (RL), and Multi-teacher On-Policy Distillation (MOPD). Nemotron 3 Ultra is our most capable model yet, employing multiple key technologies - LatentMoE, Multi Token Prediction (MTP), NVFP4 pre-training, multi-environment RLVR, MOPD, and reasoning budget control. Nemotron 3 Ultra achieves up to 6x higher inference throughput as compared to state-of-the-art publicly available LLMs while attaining on-par accuracy. The state-of-the-art accuracy, high inference throughput, and 1M token context length make Nemotron 3 Ultra ideal for long-running autonomous agentic tasks. We open-source the base, post-trained, and quantized checkpoints, along with the training data and recipe on HuggingFace.
Sources
- gpt-oss-120b & gpt-oss-20b Model Card
- OpenCodeReasoning: Advancing Data Distillation for Competitive Coding
- MIND: Math Informed syNthetic Dialogues for Pretraining LLMs
- Program Synthesis with Large Language Models
- SWE-rebench: An Automated Pipeline for Task Collection and Decontaminated Evaluation of Software Engineering Agents
- SWE-rebench V2: Language-Agnostic SWE Task Collection at Scale
- $\tau^2$-Bench: Evaluating Conversational Agents in a Dual-Control Environment
- Finance Agent Benchmark: Benchmarking LLMs on Real-world Financial Research Tasks
- PIQA: Reasoning about Physical Commonsense in Natural Language
- NVIDIA Nemotron 3: Efficient and Open Intelligence
- Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
- DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models
- DeepSeek-V3 Technical Report
- Beyond Benchmarks: MathArena as an Evaluation Platform for Mathematics with LLMs
- ACE-RTL: When Agentic Context Evolution Meets RTL-Specialized LLMs
- Nemotron-Math: Efficient Long-Context Distillation of Mathematical Reasoning from Multi-Mode Supervision
- Maximize Your Data's Potential: Enhancing LLM Accuracy with Two-Phase Pretraining
- Towards Autonomous Mathematics Research
- GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Models
- Measuring Massive Multitask Language Understanding
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering