ClayBuddy: A Framework, Evaluation, & Mitigation of Coding Agent Failures

arXiv:2606.19380 · cs.SE, cs.LG · Submitted 2026-06-13 · Read on arXiv

cs.SE, cs.LG

Submitted: 2026-06-13

Updated: 2026-08-26

Comments: Accepted at The Third Annual Conference on Language Modeling 2026 Workshop on Agent Behavior

Code: https://github.com/kenneth-ge/claybuddy

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Terminology

• Accuracy — 17× in this paper · explained in 1 episode
• Calibration — 16× in this paper · explained in 7 episodes
• Agent Harness — 13× in this paper · explained in 2 episodes
• Affordance — 12× in this paper · explained in 1 episode
• Failure Modes — 8× in this paper · explained in 3 episodes
• Alignment — 7× in this paper · explained in 1 episode
• Benchmark — 7× in this paper · explained in 2 episodes
• Trace — 7× in this paper · explained in 2 episodes
• AI Agents — 5× in this paper · explained in 2 episodes
• Core — 5× in this paper · explained in 2 episodes
• AUC — 4× in this paper · explained in 2 episodes
• Ablation — 3× in this paper · explained in 1 episode
• Adaptation — 3× in this paper · explained in 1 episode
• Consistency — 3× in this paper · explained in 3 episodes
• Pruning — 3× in this paper · explained in 4 episodes
• Actor — 2× in this paper · explained in 3 episodes
• Adversarial scenarios — 2× in this paper · explained in 1 episode
• Embeddings — 2× in this paper · explained in 2 episodes
• Scaffolding — 2× in this paper · explained in 3 episodes
• Transformer — 2× in this paper · explained in 2 episodes
• alpha — 2× in this paper · explained in 1 episode
• Abundance — 1× in this paper · explained in 1 episode
• Adaptability — 1× in this paper · explained in 1 episode
• Agentic Coding — 1× in this paper · explained in 1 episode
• Agentic Evaluation — 1× in this paper · explained in 1 episode
• Arbiter — 1× in this paper · explained in 1 episode
• Benchmarking — 1× in this paper · explained in 3 episodes
• Commentary — 1× in this paper · explained in 2 episodes
• Generative Model — 1× in this paper · explained in 2 episodes
• Human-in-the-Loop — 1× in this paper · explained in 3 episodes
• LLM judge — 1× in this paper · explained in 2 episodes
• LLM-as-a-Judge — 1× in this paper · explained in 3 episodes
• Latent Reasoning — 1× in this paper · explained in 2 episodes
• Perplexity — 1× in this paper · explained in 3 episodes
• Self-consistency — 1× in this paper · explained in 2 episodes
• Steering — 1× in this paper · explained in 2 episodes
• Two-Stage Architecture — 1× in this paper · explained in 2 episodes
• Verification — 1× in this paper · explained in 2 episodes

Sources

• AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents
• Check Yourself Before You Wreck Yourself: Selectively Quitting Improves LLM Agent Safety
• Operationalizing Contextual Integrity in Privacy-Conscious Assistants
• The Hot Mess of AI: How Does Misalignment Scale With Model Intelligence and Task Complexity?
• Lost in the Middle: How Language Models Use Long Contexts
• Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations
• Towards a Science of AI Agent Reliability
• Contextual Agent Security: A Policy for Every Purpose
• SafeArena: Evaluating the Safety of Autonomous Web Agents
• R-Judge: Benchmarking Safety Risk Awareness for LLM Agents
• Agent-SafetyBench: Evaluating the Safety of LLM Agents

Related papers

• Falsification-Based Verification of LLM-Generated Optimization Models: Sound Test Batteries and Their Detection Limits
• GitSkills: A Dataset of Agent Skills on GitHub
• For Coding Agents, the Repository Matters
• SABER: Benchmarking Operational Safety of LLM Coding Agents in Stateful Project Workspaces
• PackMonitor: Enabling Zero Package Hallucinations Through Decoding-Time Monitoring
• IntentCoding: Amplifying User Intent in Code Generation