Contain, Conform, Compare (Extended Version): A Red-Teamed Evaluation Kit for Small Coding Agents and the Full Record of Its Experiments
cs.AI
Submitted: 2026-11-12
Updated: 2026-11-12
Code: https://github.com/MinhThanghus/drydock-eval
Terminology
Sources
- Lessons from the Trenches on Reproducible Evaluation of Language Models
- Gemma 4 Technical Report
- SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
- AI Agents That Matter
- Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations
- SandboxEval: Towards Securing Test Environment for Untrusted Code
- Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting
- OpenHands: An Open Platform for AI Software Developers as Generalist Agents
- SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering
- SWE-smith: Scaling Data for Software Engineering Agents
- Establishing Best Practices for Building Rigorous Agentic Benchmarks
Related papers
- Multi-Objective Structured Pruning of LLMs for Latency and Model Size Optimization
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Embeddings for Preferences, Not Semantics
- Can We Trust Item Response Theory for AI Evaluation?
- Diagnosing Task Insensitivity in Language Agents