HarnessSQL: Harness-Native Training for SQL Agents in Realistic Database Environments
cs.CL
Submitted: 2026-10-08
Updated: 2026-10-14
Code: https://github.com/YangHaolin0526/HarnessSQL
License: http://creativecommons.org/licenses/by/4.0/
The gist: Text-to-SQL models are commonly trained to map questions directly to static queries, whereas real-world database agents operate through stateful, multi-turn interaction with live databases --
Terminology
Abstract
Text-to-SQL models are commonly trained to map questions directly to static queries, whereas real-world database agents operate through stateful, multi-turn interaction with live databases -- inspecting schemas, executing probe queries, diagnosing errors, and revising hypotheses. This creates a critical train-deploy mismatch, as the execution harness that mediates this interaction is introduced only at inference time. To bridge this gap, we propose HarnessSQL, a harness-native post-training framework that preserves the full interaction structure throughout both supervised fine-tuning and reinforcement learning. HarnessSQL builds isolated, executable database environments paired with hidden execution oracles, rolls out teachers directly inside the target SQL harness, and retains only verified trajectories for full-sequence SFT, followed by execution-reward RL. Across Spider 2.0-SQLite, HarnessSQL dramatically boosts the execution accuracy of compact models, raising Qwen3-8B from 15.5% to 45.2% and Qwen3-14B from 22.2% to 54.8%, while transferring effectively to out-of-distribution interactive benchmarks such as BIRD-Interact and LiveSQLBench. Our findings demonstrate that training database agents directly within their execution harness is essential for mastering complex, long-horizon database workflows.
Sources
- FireAct: Toward Language Agent Fine-tuning
- ReFoRCE: A Text-to-SQL Agent with Self-Refinement, Consensus Enforcement, and Column Exploration
- BIRD-INTERACT: Re-imagining Text-to-SQL Evaluation for Large Language Models via Lens of Dynamic Interactions
- SWE-SQL: Illuminating LLM Pathways to Solve User SQL Issues in Real-World Applications
- ToolACE: Winning the Points of LLM Function Calling
- AgentBench: Evaluating LLMs as Agents
- APIGen: Automated Pipeline for Generating Verifiable and Diverse Function-Calling Datasets
- AgentInstruct: Toward Generative Teaching with Agentic Flows
- FlexSQL: Flexible Exploration and Execution Make Better Text-to-SQL Agents
- ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs
- DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
- CHESS: Contextual Harnessing for Efficient SQL Synthesis
- ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Cases
- Executable Code Actions Elicit Better LLM Agents
- Does Synthetic Layered Design Data Benefit Layered Design Decomposition?
- MTIR-SQL: Multi-turn Tool-Integrated Reasoning Reinforcement Learning for Text-to-SQL
- $\tau$-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains
- Harness-Bench: Evaluating Agent Harnesses across Models and Realistic Workflows
- AgentTuning: Enabling Generalized Agent Abilities for LLMs
- Group Sequence Policy Optimization
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering