Motif 3: Technical Report

Motif 3 is a decoder-only Mixture-of-Experts (MoE) language model with 314 billion total parameters and 13.2 billion parameters activated for each token. Each sparse MoE layer contains 384 routed experts, while only eight are selected for each token. This fine-grained sparsity provides substantial expert capacity while limiting the expert computation required for each token. Motif 3 is built around Grouped Differential Latent Attention (GDLA), which integrates grouped differential attention with the compressed key-value representation of Multi-head Latent Attention. The architecture further incorporates modified manifold-constrained hyper-connections, Expert-Specific PolyNorm activations, and multi-token prediction to improve optimization stability, expert specialization, and inference efficiency. We pretrain Motif 3 on approximately 12.5 trillion tokens spanning web documents, STEM, code, mathematics, multilingual content, and domain-specialized corpora. A collection of expert-balancing and numerical-stabilization techniques supports stable training at scale, while selective MXFP8 computation and communication, memory-efficient fused kernels, and window-aware context parallelism enable training with context lengths up to 256K tokens. Our post-training pipeline combines general supervised fine-tuning, six specialist teachers trained with reinforcement learning, a software-engineering teacher trained with supervised fine-tuning, and Multi-teacher On-Policy Distillation. The resulting unified model consolidates complementary capabilities in reasoning, coding, tool use, professional work, long-context understanding, calibrated abstention, and instruction following. Across a broad evaluation suite, Motif 3 demonstrates competitive performance against leading open-weight models, including strong results on long-horizon agentic tasks, mathematical reasoning, scientific knowledge, and hallucination-sensitive evaluation.

The model is composed of repeated Transformer blocks, each containing a GDLA layer followed by a sparse MoE feed-forward layer. A hybrid GDLA schedule repeats one full causal-attention layer followed by three sliding-window attention layers. GDLA combines Grouped Differential Attention with Multi-head Latent Attention and applies a query-dependent output gate. The sparse MoE feed-forward layers use Expert-Specific PolyNorm activations. Multi-token prediction is adopted following DeepSeek-V3.

GDLA retains differential noise suppression, uses asymmetric GDA allocation for efficient signal modeling, and preserves MLA’s compact inference state. It applies a single normalization and up-projection to the compressed KV latent and separates the signal and noise paths only along the expanded head dimension. In a controlled training comparison, GDLA maintains a lower loss than both GDA and MLA and reaches a loss of 3.2 with 9.2% fewer training tokens than MLA.

The query is constructed through a low-rank down-projection and up-projection. The KV down-projection jointly produces a compressed KV representation and a decoupled rotary key. The full KV latent is normalized and expanded through a single projection into 16 expanded content-key and value heads shared by the signal and noise query paths. RoPE is applied only to the rotary query and key components. The signal and noise paths independently compute attention, with the signal path containing g times as many heads. A token-dependent coefficient λt = σ(xt Wλ) ∈ (0,1) nS is predicted for every signal head and scales the noise term before subtraction. The grouped differential output is Dt = HS,t − λt ⊙ Repeatg(HN,t). Before the final output projection, the differential representation is modulated by an element-wise sigmoid gate computed from the normalized query latent.

Motif 3 replaces conventional residual additions with a modified form of manifold-constrained hyper-connections (mHC). mHC generalizes a single residual stream into multiple parallel streams and learns token-dependent mappings that select, transform, and redistribute information across them. Motif 3 uses an expansion rate of n = 4, corresponding to four parallel residual streams. The pre-mapping and post-mapping are nonnegative vectors, and the residual mapping is constrained to the Birkhoff polytope, so its entries are nonnegative and every row and column sums to one. The post-mapping uses a time-dependent scale st that is gradually reduced from the original multiplier of two to one during pretraining to prevent repeated residual amplification from producing activation outliers at large depth.

Each sparse MoE layer contains 384 routed experts and one shared expert, with eight routed experts selected for each token. Motif 3 replaces the SiLU gate activation with Expert-Specific PolyNorm, whose polynomial coefficients and bias are learned independently for each expert. The activation is PolyNormi(z) = Σ n=1 cubed a i,n z n / RMS(z n) + b i. A sigmoid parameterization constrains each polynomial coefficient to (0,1), while the bias is clipped to [−0.5, 0.5]. Expert-Specific PolyNorm preserves a higher effective rank in the expert gate weights than SwiGLU across the measured layers.

The model configuration includes 53 Transformer layers with a model dimension of 4,096. The first two layers use dense feed-forward networks with an intermediate dimension of 12,288, while the remaining 51 layers use sparse MoE feed-forward networks. Each MoE layer contains 384 routed experts and one shared expert. The router uses normalized sigmoid scores and selects eight routed experts per token. Each routed expert has an intermediate dimension of 1,280. GDLA uses 80 query heads and 16 KV heads, with a query-key head dimension of 192, a value-head dimension of 128, and a 64-dimensional rotary component. The query and KV low-rank dimensions are 1,024 and 512, respectively. A grouped ratio of g = 4 yields 64 signal and 16 noise query heads. Every fourth layer uses full causal attention, and the remaining layers use sliding-window attention with a window of 128 tokens.

Training uses a hierarchical parallel layout with expert parallelism (EP = 8) as the innermost dimension, a sharded data-parallel degree of 8, and replicated data-parallel for remaining ranks. Pipeline parallelism is not used. Operator-level selective activation recomputation is used throughout pre-training and for SFT at sequence lengths up to 128K, with full activation recomputation at 256K tokens. The output projection and cross-entropy are computed jointly using the Liger fused linear cross-entropy kernel. Gradient Reduce-Scatter operations are overlapped with backward computation using a bounded window of in-flight operations. Gradient synchronization across DP-replicate ranks is suppressed for intermediate microbatches and issued only for the final microbatch. Expert weights are not resharded after backward on intermediate microbatches, and for a selected subset of MoE layers (22 of 51), experts remain unsharded between forward and backward.

MXFP8 is applied to the MoE experts, with DeepGEMM as the primary grouped GEMM backend. Expert activations are quantized before EP dispatch so that the dispatch communication also benefits from reduced precision. Gradient synchronization uses BF16 for communication while performing reductions locally in FP32. Row-wise MXFP8 weights are All-Gathered, and the column-wise copy is produced locally by a fused row-to-column transcode kernel.

Muon is used as the primary optimizer, with AdamW for embeddings, the output projection, and vector-shaped parameters. Parallel Muon is used for MoE parameters, with expert weights partitioned along the expert dimension. QK-Clip is applied to control attention-logit growth, with an asymmetric split between query and key scaling to prevent the shared key projection from collapsing. Optimizer states are offloaded to host memory after every optimizer step and reloaded layer by layer in step with backward computation.

Several operators are fused into single custom kernels, including a pad-aware fused GroupedPolyNorm kernel, mHC kernels, and a fused PolyNorm–FP8 inference kernel.

For long-context training with sequences up to 256K tokens, context parallelism (CP = 8) reuses the same process dimension as EP. The context-parallel algorithm is selected per layer: sliding-window layers use window-aware Ring Attention, while full-attention layers use Ulysses. Window-aware Ring Attention reduces the per-rank received KV volume from Θ(L) to Θ(W), an analytic factor of 1,024× at the 128-token operating window. Attention-workload-aware re-scheduling rebalances complete packed sequences across the data-parallel dimension using a deterministic longest-processing-time greedy heuristic.

Pretraining data consists of approximately 12.5 trillion tokens from a broad mixture including general web documents, STEM material, source code, synthetic question-answer pairs, mathematical content, Korean and multilingual corpora, reasoning-intensive examples, and legal and financial domain data. The retained Nemotron data account for approximately 70% of the full pretraining corpus. Dynamic data-mixture scheduling is used, expanding from 19 to 57 constituent datasets over training. The tokenizer is trained from scratch using SuperBPE, which learns superword tokens spanning multiple whitespace-delimited words. Motif 3 achieves the strongest compression on English, Korean, code, and mathematics compared to recent open-source tokenizers.

MoE training stabilization includes FP32 routing and selective load balancing, decaying router noise, FFN magnitude regularization, and expert-health monitoring. A sequence-wise auxiliary load-balancing loss is applied with stronger weights for the earliest MoE layers. Router noise follows a cosine decay schedule. The FFN magnitude regularizer applies a soft auxiliary penalty to each FFN output. Expert-health monitoring tracks dispatch min/median, maximum expert-token count, output-weight min/median, routed/shared RMS ratios, maximum expert-output cosine similarity, and routed and shared abs-max/RMS.

The global training batch contains up to 75 million tokens. Pretraining begins with sequences of up to 4K tokens, transitions to 32K tokens during the learning-rate decay phase, followed by a dedicated long-context stage with sequences of up to 256K tokens. The full-attention layers use a DeepSeek-YaRN extension from the original 4K context length to 256K, with an extension factor of 64. Training starts with a learning rate of 3 × 10−4 and follows a warmup-stable-decay schedule, reducing to 7.5 × 10−5 during the decay phase and 3 × 10−5 for the long-context stage. The MTP auxiliary objective uses a loss weight of 0.2. The expert-selection bias uses a balancing coefficient of 1 × 10−3. The sequence-wise auxiliary load-balancing loss uses a default weight of 1 × 10−4, increased to 2.4 × 10−4 for layer 2 and 2.0 × 10−4 for layers 3–5. The FFN magnitude regularizer uses a loss weight of 2.0 × 10−4 and a threshold of 128, with a threshold of 1,024 for the final layer.

The pretrained base model achieves 86.20 on MMLU (5-shot), 68.56 on MMLU-Pro (5-shot CoT), 94.71 on ARC-C (25-shot), 80.90 on WinoGrande (5-shot), 88.30 on HellaSwag (10-shot), 85.14 on PIQA (0-shot), 93.93 on GSM8K (8-shot CoT), 70.58 on MATH (4-shot CoT), 73.70 on HumanEval (0-shot), and 84.60 on MBPP (3-shot).

Post-training consists of three stages: general supervised fine-tuning (SFT), specialized teacher training, and Multi-teacher On-Policy Distillation (MOPD). The SFT corpus is constructed primarily from selected datasets in the Nemotron family, supplemented with broadly collected and synthetically generated data. A preliminary SFT model identifies failure-prone decisions in unsuccessful agentic trajectories and guides targeted data generation. The final student is trained from the pretrained checkpoint on the consolidated corpus. SFT uses a global batch size of 33 million tokens, a learning rate of 3 × 10−5 decaying to 1.5 × 10−6, and a maximum sequence length of 256K tokens. The expert-selection bias coefficient and sequence-wise auxiliary load-balancing loss weights are each reduced by a factor of ten. The MTP objective uses a loss weight of 0.1. Assistant-only loss masking is applied to both the standard next-token prediction objective and the MTP objective.

For reinforcement learning, TorchTitan is implemented as an external NeMo RL training backend, allowing the same Motif 3 implementation and checkpoints to be optimized directly with NeMo RL’s GRPO algorithm. Asynchronous GRPO with in-flight weight updates is used, with a one-step trajectory age bound and truncated importance-sampling correction. Six specialist teachers are trained with GRPO across 13 verifier domains, and a seventh software-engineering teacher is trained with SFT. The teachers cover agentic tool use, professional work, software engineering, long-context reasoning and abstention, mathematics, code and science, and chat. Frozen MTP and routing parameters are maintained throughout specialist-teacher training and MOPD. Offline prompt filtering retains prompts with empirical pass rates satisfying 0 < p ≤ 0.8.

The agentic tool-use teacher trains on end-to-end terminal tasks with task-specific outcome verifiers and no intermediate process supervision. The professional-work teacher trains on long-horizon workplace tasks with preference-based rewards from an LLM judge. The software-engineering teacher is trained with SFT on repository-level tasks, retaining only successful model-generated trajectories that pass tests. The long-context reasoning and abstention teacher uses a deterministic LLM judge with a four-grade reward ladder. The mathematics teacher verifies answers with symbolic and structured mathematical checking libraries. The code-and-science teacher uses coding tasks with task-specific test cases and direct execution verification. The chat teacher targets dialogue quality, instruction following, and safety, with a two-stage reward for Korean chat tasks.

MOPD consolidates the capabilities of all seven teachers into the general SFT student. On-policy trajectories are generated with the student, each example is routed to its domain specialist, and environment grading is disabled. For each token sampled by the student, only the scalar log probability assigned to that token by the routed teacher is evaluated. An ICE-POP filter drops tokens whose importance weights fall outside [0.5, 5.0]. The distillation loss is LMOPD = −E t[w̃ t d t log π θ(y t x, y<t)], where d t is the detached OPD signal. A global batch of 512 on-policy trajectories is used, with the learning rate decaying from 5 × 10−6 to 3 × 10−6.

Evaluation results show Motif 3 performs particularly well on agentic and tool-oriented benchmarks. It achieves 38.7 on GDPval-AA v2, 94.7 on τ2-Bench Telecom, 35.3 on τ3-Banking, and 51.5 on the public subset of ITBench-AA. On coding benchmarks, it achieves 76.2 on SWE-bench Verified, 74.9 on Terminal-Bench 2.1, and 40.6 on SciCode. On reasoning and knowledge benchmarks, it scores 83.2 on IMO-AnswerBench, 75.5 on Apex Shortlist, 83.4 on GPQA Diamond, 37.0 on HLE, 6.6 on CritPt, 30.1 on AA-Omniscience Accuracy, and 71.6 on AA-Omniscience Non-Hallucination. On long-context and instruction-following benchmarks, it achieves 72.3 on AA-LCR and 78.2 on IFBench.

The model has several limitations. Its training and evaluation do not cover the full diversity of real-world tasks, domains, languages, interaction patterns, and deployment conditions. It is primarily a text model, limiting applicability to tasks requiring direct understanding of visual inputs. Many long-horizon applications require reliable state tracking, planning, recovery, and environment interaction over substantially longer trajectories than those evaluated. Future work will investigate new architectures, extend native context lengths beyond one million tokens, add visual capabilities, and strengthen long-horizon agent capabilities.

Improvements to AI systems:

1. Implement Grouped Differential Latent Attention (GDLA) as a drop-in replacement for standard attention in decoder-only models. This reduces training tokens needed to reach a given loss by 9.2% (e.g., loss 3.2) while maintaining compact inference state via latent KV compression, enabling longer context with lower memory.

2. Adopt Expert-Specific PolyNorm activations in MoE feed-forward layers instead of SiLU/SwiGLU. This preserves higher effective rank in expert gate weights, improving expert specialization and reducing capacity waste, leading to better per-expert utilization and more accurate routing.

3. Use modified manifold-constrained hyper-connections (mHC) with 4 parallel residual streams and Birkhoff-polytope-constrained mappings. This replaces single residual streams, enabling token-dependent information redistribution across streams, which stabilizes deep training and reduces activation outliers, allowing more layers (e.g., 53) without degradation.

4. Integrate multi-token prediction (as in DeepSeek-V3) with a loss weight of 0.2 during pretraining and 0.1 during SFT. This improves sample efficiency and inference speed by enabling speculative decoding, reducing latency for autoregressive generation.

5. Deploy a hybrid attention schedule (1 full causal attention per 3 sliding-window layers, window=128) combined with window-aware Ring Attention for context parallelism. This reduces per-rank KV communication from O(L) to O(W)—a 1,024× reduction at 128-token window—enabling efficient training on 256K-token sequences with modest hardware.

6. Apply asymmetric QK-Clip with separate scaling for query and key projections. This prevents attention-logit explosion and avoids collapse of the shared key projection, improving training stability at scale and enabling reliable long-context extension (4K to 256K via DeepSeek-YaRN).

7. Use Muon optimizer for MoE parameters with parallel partitioning along the expert dimension, while keeping AdamW for embeddings and output projections. This accelerates convergence for large sparse models and reduces optimizer memory overhead via host offloading.

8. Implement selective MXFP8 computation for MoE experts, including quantizing expert activations before EP dispatch. This reduces communication bandwidth and compute cost by 2× while maintaining accuracy through row-wise MXFP8 weights and fused row-to-column transcode kernels.

9. Adopt a multi-teacher on-policy distillation (MOPD) framework with 7 specialist teachers (agentic tool use, professional work, software engineering, long-context reasoning/abstention, math, code/science, chat) trained via GRPO and SFT. The student model distills all capabilities using token-level importance weighting with ICE-POP filter (weights in [0.5, 5.0]), enabling a single unified model to excel across diverse tasks without catastrophic forgetting.

10. Use sequence-wise auxiliary load-balancing loss with layer-specific weights (e.g., 2.4e-4 for layer 2, 2.0e-4 for layers 3–5) and decaying router noise (cosine schedule). This prevents expert collapse and ensures balanced token routing, critical for stable training of 384-expert MoE layers.

11. Apply FFN magnitude regularization with a soft penalty (weight 2e-4, threshold 128) to prevent output explosion in deep MoE layers, improving numerical stability and enabling higher learning rates.

12. Employ SuperBPE tokenizer trained from scratch, which learns superword tokens spanning multiple whitespace-delimited words. This achieves stronger compression on English, Korean, code, and math, reducing sequence length and inference cost.

What the improved AI system can do:

• Train faster and cheaper: Achieve state-of-the-art loss with ~9% fewer tokens, reducing pretraining compute and cost.

• Handle ultra-long context (up to 256K tokens): Efficiently process entire codebases, long documents, or multi-turn agentic trajectories with linear memory scaling via window-aware attention.

• Excel at agentic and tool-use tasks: Achieve 38.7 on GDPval-AA, 94.7 on τ2-Bench Telecom, and 51.5 on ITBench-AA, enabling reliable end-to-end terminal interaction, API calling, and multi-step planning.

• Solve complex software engineering problems: Score 76.2 on SWE-bench Verified and 74.9 on Terminal-Bench 2.1, allowing autonomous bug fixing, repository-level code generation, and test-driven development.

• Reason at expert level in math and science: Score 83.2 on IMO-AnswerBench, 83.4 on GPQA Diamond, and 40.6 on SciCode, supporting symbolic verification and structured scientific reasoning.

• Provide calibrated abstention: Achieve 71.6 on AA-Omniscience Non-Hallucination, enabling the model to refuse uncertain answers, reducing hallucination in high-stakes domains.

• Maintain low inference cost: With 13.2B active parameters out of 314B total and compact KV cache from GDLA, serve at scale with reduced latency and memory.

• Support multilingual and domain-specialized tasks: Strong performance on Korean and legal/financial data, with robust instruction following (78.2 on IFBench) and long-context reasoning (72.3 on AA-LCR).

• Stable training at 12.5T tokens: Use expert-health monitoring, FP32 routing, and selective load balancing to avoid divergence, enabling reproducible large-scale pretraining.