Benchmark-Driven Selection of AI: Evidence from DeepSeek-R1

arXiv:2508.10173 · cs.LG, cs.CY · Submitted 2025-08-13 · Read on arXiv

cs.LG, cs.CY

Submitted: 2025-08-13

Updated: 2025-08-13

Code: https://github.com/sambowyer/bayes_evals

Terminology

• Benchmark — 5× in this paper · explained in 2 episodes
• Trace — 2× in this paper · explained in 2 episodes
• Algorithmic Improvements — 1× in this paper · explained in 1 episode
• Generalization — 1× in this paper · explained in 13 episodes
• Steering — 1× in this paper · explained in 2 episodes

Sources

• Phi-4-reasoning Technical Report
• MathArena: Evaluating LLMs on Uncontaminated Math Competitions
• DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
• A Sober Look at Progress in Language Model Reasoning: Pitfalls and Paths to Reproducibility
• OpenAI o1 System Card
• Tulu 3: Pushing Frontiers in Open Language Model Post-Training
• ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models
• Humanity's Last Exam
• Show Your Work: Scratchpads for Intermediate Computation with Language Models
• OMEGA: Can LLMs Reason Outside the Box in Math? Evaluating Exploratory, Compositional, and Transformative Generalization
• Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning

Related papers

• Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
• AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
• Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
• Convergence issues in Relational Concept Analysis based on AOC-posets
• Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
• Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks