One Run Is Not Enough: A Monte Carlo Protocol for Evaluating LLM Reliability on Data Science Tasks
cs.LG
Submitted: 2026-11-06
Updated: 2026-11-06
Code: https://github.com/niwq15/LLM-DS-Reliability
Terminology
Sources
- FrontierMath: A Benchmark for Evaluating Advanced Mathematical Reasoning in AI
- Text2Analysis: A Benchmark of Table Question Answering with Advanced Data Analysis and Unclear Queries
- InfiAgent-DABench: Evaluating Agents on Data Analysis Tasks
- MLAgentBench: Evaluating Language Agents on Machine Learning Experimentation
- LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code
- DS-1000: A Natural and Reliable Benchmark for Data Science Code Generation
- IMProofBench: Benchmarking AI on Research-Level Mathematical Proof Generation
Related papers
- A Physics-Regulated Neural Framework for Learning 3D Grain Growth Dynamics
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis