SiriusDeliver: Automating Data Warehouse Delivery at Tencent

SiriusDeliver is an end-to-end delivery automation agent for production warehouse task submission at Tencent, designed to address the limitations of general-purpose coding agents in enterprise data warehouse (DW) environments. The paper states that warehouse task delivery remains a largely manual process involving context retrieval, workflow configuration, code generation, platform submission, and failure diagnosis, and that SQL development accounts for less than 25% of end-to-end delivery time, while information retrieval and cross-platform coordination collectively constitute the primary sources of effort.

The system addresses three key challenges: (1) multi-stage task composition, where one-shot LLM planning frequently omits or misorders prerequisites; (2) trustworthy artifact delivery, where syntactically valid artifacts may still contain invalid dependencies, schedule conflicts, or logic bugs; and (3) continuous platform adaptation, as platform APIs, data schemas, scheduling policies, and operational conventions continually evolve.

SiriusDeliver integrates three core components:

• Delivery automation agent: maintains a structured delivery state and orchestrates warehouse skills to incrementally construct a candidate artifact bundle, organizing skills into four layers (scenario, context, artifact, platform) and using a manager, executor, and memory to iteratively plan, route, execute, and update.

• Artifact lifecycle control: validates and revises artifacts in two complementary stages—pre-execution diagnosis using deterministic platform evidence with LLM-based reasoning (evidence grounding, hybrid verification, candidate revision) and post-execution diagnosis that extracts failure signals, retrieves diagnostic knowledge, and performs root-cause-based revision.

• Trace-driven skill evolution: converts delivery trajectories into bounded and reviewable skill refinements through evidence preparation, experience-guided refinement (with self-guided and external-guided experience), and validation/deployment with offline replay validation and mandatory human review.

Offline evaluation on 200 real-world delivery cases across 4 warehouse scenarios shows SiriusDeliver improves the average end-to-end delivery success rate by 14.5 points (from 71.5% to 86.0%) over the strongest skill-augmented baseline and reduces token consumption by 30%. Ablation studies show removing hierarchical skill orchestration causes the largest degradation, an 11.5-point success drop (86.0%→74.5%); removing artifact lifecycle control reduces success by 8.0 points; removing trace-driven skill evolution causes a 2.0-point drop but increases token consumption from 86.9K to 112.3K.

The system was deployed on Tencent Cloud WeData for two months across 6 business teams and 4 warehouse task types, serving 3,600 monthly active users and supporting 18,240 delivery sessions, achieving an 87.2% end-to-end success rate and a 73.5% autonomous submission rate. A one-month A/B test with 70 engineers (35 per group) showed SiriusDeliver reduced the median end-to-end delivery time from 228 minutes to 23 minutes (89.9% reduction), net engineer effort from 95 to 11 minutes (88.4% reduction), time to first executable artifact from 44 to 2.6 minutes, and manual intervention rate from 100% to 21%, while final delivery success rates remained comparable between the treatment and control groups, 97.6% vs. 98.1%.

Additional production insights include: artifact lifecycle control was triggered in 61% of sessions: pre-execution diagnosis detected 4,120 risky artifacts before submission, and post-execution diagnosis repaired 2,890 platform execution failures; trace-driven skill evolution produced 128 human-approved skill updates, including 89 repair-oriented updates and 39 compression-oriented updates; and user ratings averaged 4.4 out of 5.0 from 3,180 ratings.

Improvements to AI Systems:

1. Hierarchical skill orchestration with explicit state tracking – Instead of one-shot planning, the AI maintains a structured delivery state and iteratively plans/routes/executes across layered skill modules (scenario → context → artifact → platform). This reduces omission and misordering of prerequisites in multi-step tasks. The improved system can handle complex, multi-stage workflows (e.g., data pipeline construction, CI/CD deployment) by breaking them into verifiable sub-tasks and re-planning dynamically.

2. Dual-stage artifact validation with deterministic grounding – The AI combines pre-execution checks (using deterministic platform evidence like schema constraints, dependency graphs, and schedule conflicts) with LLM-based reasoning to catch invalid dependencies or logic bugs before submission. Post-execution, it extracts failure signals and retrieves diagnostic knowledge to perform root-cause-based revision. The improved system can self-correct its outputs (code, configs, workflows) by validating against real platform rules and learning from execution failures, reducing silent errors and rework.

3. Trace-driven skill evolution with human-in-the-loop – The AI converts successful/failed delivery trajectories into bounded, reviewable refinements of its own skills (e.g., repair-oriented or compression-oriented updates). It uses offline replay validation and mandatory human review before deployment. The improved system can continuously adapt to evolving APIs, schemas, and policies without catastrophic forgetting, while maintaining trust through human oversight. This enables long-term autonomy in changing production environments.

4. Evidence-grounded reasoning for failure diagnosis – Instead of purely generative debugging, the AI grounds its reasoning in concrete platform signals (logs, error codes, state diffs) and retrieves past diagnostic knowledge for similar failures. The improved system can diagnose and repair execution failures faster and more accurately, reducing manual intervention and improving autonomous submission rates.

5. Token-efficient incremental construction – By building artifact bundles incrementally and compressing redundant skill steps, the AI reduces token consumption by 30% while maintaining or improving success rates. The improved system can handle longer, more complex tasks within budget constraints, making it feasible for large-scale enterprise use.

What the improved AI system can do:

• Automate end-to-end delivery of complex, multi-step artifacts (e.g., data warehouse tasks, ML pipelines, cloud deployments) with a success rate above 86–87% in production, compared to 71% for baseline agents.

• Reduce end-to-end task completion time by 90% (e.g., from 228 to 23 minutes) and net human effort by 88%, while keeping final quality comparable to manual expert work (97–98% success).

• Self-diagnose and repair failures before and after execution, catching thousands of risky artifacts pre-submission and fixing thousands of execution failures autonomously.

• Continuously improve its own operational skills via trace-driven learning, with human-approved updates that make it more efficient and reliable over time.

• Operate at scale (e.g., 18,000+ delivery sessions, 3,600 monthly users) with high user satisfaction (4.4/5 rating) and a 73.5% fully autonomous submission rate, requiring human intervention only in 21% of cases.