Carnot: Interpretable, Interactive, and Optimized Execution of Deep Research Queries

Carnot is an interactive execution engine for AI-driven analytics that compiles natural language requests into physical execution graphs and surfaces them through an interactive notebook interface. It addresses the problem that Deep Research systems operate as opaque black boxes, hiding intermediate reasoning and data retrieval steps, and failing to expose controls for managing API costs and execution latency, while semantic query processing engines can be prohibitively expensive for enterprise-scale data lakes. Carnot provides both visibility—the ability to inspect every prompt, code block, and intermediate result before, during, and after execution—and steerability—the ability to intervene at any step without discarding prior computation.

When a user issues a natural language query, Carnot’s automated planner compiles it into a Directed Acyclic Graph (DAG) of physical operators, combining semantic operators (e.g., LLM-based filters and extractors) with agentic operators (e.g., data exploration, Python code generation and execution). Unlike prior semantic operator systems, Carnot materializes the DAG as a notebook interface where each cell corresponds to a single operator. Users can inspect any cell at multiple levels of abstraction—a summary of the operator’s intent, the generated code or prompt, or the raw input/output data pairs—and execute the graph one operator at a time. When a user edits a cell to correct a mistake, Carnot’s optimizer will automatically re-optimize and re-execute the cell while reusing the results of unaffected upstream cells.

The frontend includes a data management page for uploading raw data files and organizing them into named datasets, a settings page for registering API keys for LLM providers, and a chat interface where users issue queries. The backend manages metadata and state in a Postgres database, stores user-uploaded data files in a private S3 bucket, and dispatches planning, optimization, and execution requests to the Carnot library. The library provides modular components including a Dataset class with chainable operators, a QueryPlanner that is an LLM agent with few-shot demonstrations, a QueryOptimizer inspired by the Cascades-style optimizer of Abacus that uses an LLM agent to select the final physical plan from the estimated Pareto frontier of cost, latency, and quality, and a streaming and notebook execution layer with a storage engine supporting caching and an index catalog.

The demonstration showcases three scenarios: transparent plan inspection where attendees issue a free-form query and examine the DAG of operators before any computation begins; steering via the chat interface where attendees provide follow-up instructions that cause the planner to re-plan affected operators; and editing via the notebook interface where attendees execute cells step-by-step, edit a cell to fix a faulty semantic filter, and observe Carnot re-optimize and resume execution without re-running upstream cells. The demo also highlights cost-aware optimization where attendees configure explicit cost and latency constraints and see the planner dynamically swap operator implementations—e.g., replacing a large frontier model with a smaller, cheaper one for a low-precision filter—to meet the specified budget.

Carnot does not provide guarantees on plan accuracy or correctness, though the authors note that statistical guarantees with respect to an oracle could be added as a soft guarantee. Future work includes extending Carnot to support web search and general tool calling, and improving the notebook interface to better highlight the non-linear nature of most plan DAGs.

Improvements to AI Systems Based on Carnot:

1. Transparent Execution Graphs for All AI Agents

• Improvement: Force every AI agent (not just analytics) to compile its internal reasoning into an explicit, inspectable Directed Acyclic Graph (DAG) of atomic operations (e.g., retrieval, filtering, generation, tool calls).

• Improved AI capability: Users can see exactly what steps the AI took, why, and in what order—before, during, and after execution—eliminating black-box behavior.

2. Interactive Steerability with Incremental Re-optimization

• Improvement: Add a notebook cell abstraction to any AI pipeline, where each step is a modifiable unit. When a user edits a step, the AI automatically re-plans only the downstream operations while reusing cached results from unaffected upstream steps.

• Improved AI capability: Users can correct mistakes mid-task (e.g., use a stricter filter here) without restarting the entire process, saving time and compute.

3. Cost-Latency-Quality Pareto-Aware Planning

• Improvement: Integrate a multi-objective optimizer that, given user-defined budgets (e.g., max API cost, max latency), selects the best operator implementation from a Pareto frontier—e.g., swapping a large LLM for a smaller one on low-precision subtasks.

• Improved AI capability: AI systems become budget-aware, dynamically trading off accuracy for speed or cost based on explicit user constraints, rather than always using the most expensive model.

4. Multi-Level Abstraction Inspection

• Improvement: Allow users to inspect any AI operation at three levels: (a) a natural-language summary of intent, (b) the actual code/prompt executed, and (c) raw input/output data pairs.

• Improved AI capability: Users can audit AI decisions at their preferred depth—quick overview, technical detail, or raw data—making debugging and trust-building feasible for non-experts and experts alike.

5. Reusable State and Caching Across Sessions

• Improvement: Implement a persistent storage engine with an index catalog that caches intermediate results (e.g., embeddings, filtered datasets, generated code) and reuses them across different queries or sessions when inputs are unchanged.

• Improved AI capability: AI systems avoid redundant computation, dramatically reducing cost and latency for repeated or similar tasks (e.g., re-running a query with a minor tweak).

6. Human-in-the-Loop Re-planning via Conversational Steering

• Improvement: Enable users to issue mid-execution follow-up instructions (e.g., only consider data from 2023) that cause the planner to re-plan only the affected subgraph, not the whole task.

• Improved AI capability: AI systems become collaborative, adapting to user feedback in real time without losing prior progress, making them suitable for exploratory data analysis and iterative decision-making.

7. Statistical Soft Guarantees on Output Quality

• Improvement: Add an optional oracle-check layer that, when enabled, compares a sample of AI outputs against a ground-truth oracle and reports a confidence interval for accuracy.

• Improved AI capability: AI systems can provide probabilistic guarantees on their outputs (e.g., "95% confidence that this filter has >90% precision"), enabling safer deployment in high-stakes domains like healthcare or finance.

8. Extensible Operator Library for Mixed Semantic-Agentic Workflows

• Improvement: Build a modular library of composable operators (LLM-based filters, extractors, code executors, web search, tool calls) that can be chained arbitrarily, with each operator exposing its own cost, latency, and quality metadata.

• Improved AI capability: AI systems can handle hybrid tasks (e.g., summarize this dataset, then run a Python script to compute correlations, then fetch external data to validate) in a single, steerable pipeline.

9. Automatic Plan Visualization for Non-Linear DAGs

• Improvement: Improve the notebook interface to visually highlight parallel branches, dependencies, and critical paths in the execution graph, rather than forcing a linear view.

• Improved AI capability: Users can quickly grasp complex multi-step AI workflows, identify bottlenecks, and decide where to intervene—critical for large-scale or time-sensitive tasks.

10. Dynamic Operator Substitution Based on Live Constraints

• Improvement: Allow the optimizer to swap operator implementations during execution (e.g., if a model times out, switch to a faster one; if cost exceeds budget, downgrade to a cheaper embedding).

• Improved AI capability: AI systems become resilient to runtime failures and budget overruns, self-adjusting to meet constraints without user intervention.