DrSai: an agentic AI system toward autonomous physics analysis workflows at BESIII

summary

The gist

High-energy physics (HEP) analysis traditionally demands extensive manual effort to navigate complex software environments and maintain physical rigor By enabling physicists to define high-level…

In short

DrSai is an AI multi-agent system designed to automate complex physics analysis workflows at BESIII. It uses specialized agents like Planner, Coder, and Tester to decompose high-level physics goals into executable steps. Validation showed it can handle real experimental tasks, though performance bottlenecks remain in precise tool calling.

Key concepts

Multi-Agent System (MAS)
A MAS uses several specialized AI agents that collaborate to solve a complex problem. In DrSai, agents like the Planner and Coder work together—the Planner sets the physical goal, while the Coder writes standardized analysis code—to automate end-to-end research tasks.
Hierarchical Task Management
This system breaks large physics goals into smaller, manageable sub-tasks organized in a selfsimilar structure. This ensures that dependencies are tracked correctly as complex workflows are decomposed from a single high-level objective down to executable code generation.
HepScript
HepScript is a domain-specific language created to formalize human analytical logic. It allows physicists to translate abstract physical intentions into standardized, executable sequences that the AI agents can process and implement within the analysis framework.

Terminology used across episodes

This episode discusses

The paper

DrSai: an agentic AI system toward autonomous physics analysis workflows at BESIII · Read on arXiv

Institute of High Energy Physics, CAS Institute of High Energy Physics, University of Chinese Academy of Sciences, Jilin University, Lanzhou University, Tsung-Dao Lee Institute Shanghai Jiao Tong University, Institute of Nuclear and Particle Physics School of Physics and Astronomy Shanghai Jiao Tong University Key Laboratory for Particle Astrophysics and Cosmology MOE

DOI: 10.1016/j.scib.2026.10.072

Transcript

Introduction to the show: ident: High Energy Physics: Experiment. The day in high-energy experiment: detectors, searches and limits.

Kai: I'm Kai.

Mira: And I'm Mira, here with Lev.

Kai: We're looking at the paper: "DrSai: an agentic AI system toward autonomous physics analysis workflows at BESIII".

Mira: Here's the gist of the paper. High-energy physics (HEP) analysis traditionally demands extensive manual effort to navigate complex software environments and maintain physical rigor By enabling physicists to define high-level physics goals while delegating workflow orchestration to autonomous agents, we present Dr.Sai, an AI-driven multiagent system designed to automate the end-to-end physics analysis workflow.

Kai: Dr.Sai was specifically engineered to handle real-world HEP tasks by bridging highlevel physics intent with the intricate software stacks of the BESIII experiment.

Mira: System Architecture and Agents The Dr.Sai system utilizes a Multi-Agent System (MAS) architecture integrated with a daemon-based message management system.

Kai: This MAS consists of specialized agents that collaborate to decompose and execute complex tasks, supported by base LLMs for foundational reasoning and natural language processing.

Title and authors: Mira: So we're looking at this paper called "DrSai: an agentic AI system toward autonomous physics analysis workflows at BESIII." It sounds like they've built something designed to take the heavy lifting out of analyzing data from experiments like BESIII.

Kai: Yeah, what they’re aiming for is to let physicists set a big physics goal and then the AI agents handle all the messy steps—the whole workflow—to get them results. It's about automating that end-to-end process so you don't have to manually juggle everything.

Lev: From an engineering standpoint, that means building a system where one agent talks to another, breaking down a huge task into smaller pieces, and those pieces actually run on real hardware without crashing. It’s not just about the idea; it’s about making sure the execution part is solid.

Mira: Exactly. The paper mentions they use a multi-agent system with six specialized agents—a Host, Planner, Coder, Tester, Calculator, and Reflector—all working together to manage this process. It's a very structured way to approach the analysis pipeline.

Kai: That structure is key because physics workflows are incredibly complex; you have to make sure every step depends correctly on the last one before you move on. The Planner agent seems to be central in making sure that hierarchy of tasks makes sense for a specific experiment, like cross-section measurements.

Lev: I worry about the actual execution speed and overhead when you put all those layers of reasoning and tool calling in place; real hardware needs to handle that without massive latency spikes. How much time does this whole system actually take compared to a human doing it?

Mira: The paper says they validated this by re-measuring branching fractions across ten different J/psi decay channels, and the workflow successfully automated the entire BESIII analysis chain, from the initial idea to estimating systematic uncertainties. It shows they got results that matched Monte Carlo simulations well.

Kai: That's a good point about matching MC simulations; it confirms that what you get out of this system is physically consistent with what we expect from theory and simulation benchmarks. But the real question is, how does the AI handle those systematic uncertainties, which are often tricky in real experiments?

Lev: That’s where the Calculator agent comes in, I think. It uses a fixed formula for calculating branching ratios that includes statistical and systematic error components derived directly from the input data you feed it. It’s a concrete function, not just some abstract reasoning.

Mira: Right, and the paper found that when they tested five different large language models, adding a moderate reflection allowance really improved the reliability of those sub-tasks for most of them. So it's not just one model being perfect; it's about tuning how the AI thinks about its own work.

Title and authors: Kai: That brings us to what they found is actually a bit tricky for all these models, which is that the main performance bottleneck isn't necessarily in understanding the physics information itself, but in the functional calling capability—that’s initiating and using those scientific tools or code correctly.

Lev: If tool calling is the weak spot, then fine-tuning for that syntax level mastery of professional tools seems like a huge area for future work. It suggests that just having good reasoning isn't enough if the AI can't actually write the right executable sequence to get the job done on the real data processing software.

Mira: I agree, and I think they point out that GPT-4o does pretty well at calling those tools, but it struggles when it has to synthesize those precise parameters needed for execution, which is a different kind of failure mode <ref:2604.22541#pg1>. It's not about knowing *what* to ask the tool; it’s about getting the exact syntax right for the tool to run.

Kai: So if we look at this paper, what they suggest as improvements is focusing heavily on making those tools and code generation layers much more specialized, maybe with fine-tuned models that have a deeper grasp of professional HEP software syntax. That moves beyond just general intelligence to domain-specific execution precision.

Lev: From my side, I think the biggest challenge they've highlighted is how to take all that static knowledge—the complex physics rules and experimental procedures—and transform it into executable logic that actually mirrors how a human expert makes those strategic decisions. That translation step is where the real difficulty lies for autonomous agents.

Mira: It’s about embedding domain-specific knowledge so the AI can make those high-level, expert choices, not just follow a pre-set script blindly when things get messy in the data. It moves from following instructions to actually making judgments that look like an expert's judgment.

Kai: So to wrap up this part of the paper, DrSai is presented as a viable framework for using multi-agent systems to accelerate discovery in complex experimental settings at BESIII, proving it can maintain physical consistency during the analysis chain. But they are clear that the next step requires more than just better reasoning; it needs better execution control.

Lev: And I'd add that on the hardware side, we need to see how this entire agentic loop runs under real constraints, because those constraints—like memory and clock speed—can totally change what’s feasible for these complex workflows.

Mira: So, the implication is that while AI is great at handling the workflow orchestration, achieving true autonomy means solving the problem of perfect translation from expert thought into machine-executable code with high fidelity. That's where we head next.

The paper's summary: Kai: So, to recap, this paper is about building an AI system called DrSai that’s designed to take physicists’ high-level physics goals and run the entire complex analysis workflow at a place like BESIII all by itself.

Mira: Exactly. It's not just a chatbot; it's a whole team of specialized agents, each one handling a specific part of the analysis, from planning the steps to doing the actual calculations.

Lev: From what I see on page two, they’re using this multi-agent system structure with six distinct agents collaborating through some message management system. It sounds like they've mapped out a very rigid chain of command for an analysis task.

Kai: Right, and that planning part is really interesting because they use something called a hierarchical task management system that’s inspired by fractal structures, which means they break the big physics objective down into smaller and smaller subtasks.

Mira: That decomposition is crucial because it makes tracking dependencies much cleaner when you have a massive workflow like this. It simplifies managing all those different steps in order.

Lev: I want to talk about the actual coding part for a second, because they mention the Coder agent uses configuration driven code generation based on a JSON file that specifies things like decay channels and selection criteria.

Kai: That’s where it gets interesting, because they are creating this HepScript language which is basically a domain specific language meant to translate human analytical logic directly into executable sequences for the analysis software.

Mira: It’s about formalizing the physical intent, so you take that high level goal and you turn it into a standardized set of rules that the machine can follow without ambiguity.

Lev: That brings up a real practical question: how does this system actually work with real codebases like ROOT? Because if it’s just generating text, it won't run on the actual data processing software.

Kai: Well, the paper shows they validated this by running the whole chain from a natural language objective all the way to signal extraction and even preliminary systematic uncertainty estimation in ten different J psi decay channels.

Mira: That experimental validation is important because it proves that DrSai can maintain physical consistency across these different measurement types, which is exactly what we need in real physics.

Lev: And they did a performance study on five mainstream large language models, and the results show that even with some reflection allowance, most models got more reliable for subtasks.

Kai: But the authors were very clear about where it stops working; the main bottleneck across all those tested models wasn't about how much information they understood, but actually initiating those functional calls to external tools.

Mira: So, the lesson there is that having deep reasoning isn't enough if the AI can’t correctly generate the precise parameters needed for a tool to execute its job properly.

Lev: That points toward the future work they’re suggesting: we need better expertise in tool calling and code generation, moving towards models that have a syntax level mastery of professional physics tools.

Kai: And they also mentioned how hard it is to get domain specific knowledge embedded so the AI can make those strategic decisions that look like a human expert making choices about where to focus the search.

Mira: It’s about transforming static information—the rules and procedures—into executable logic that mirrors how a real scientist makes high level decisions.

Lev: So, the next challenge isn't just making the AI smarter, it's making it better at precise execution and having a proper way to embed all that specialized physics knowledge into its thinking structure.

The paper's improvements: Kai: So, we've seen how DrSai works right now, and now we look at what they think needs to happen next to make it better than just a powerful tool for a chat assistant.

Mira: Right, the paper points out that there are two main areas they want to focus on improving: making the AI much better at calling those professional tools and improving how it handles code generation.

Lev: I see that as needing specialized synthesis layers or maybe fine-tuned models that actually have a syntax level mastery of things like ROOT or other HEP software. It’s about getting the code and parameters exactly right the first time.

Kai: That makes sense because if the AI can't generate schema compliant parameters, it just produces garbage for execution, no matter how good its high level reasoning is.

Mira: And then there’s that second big piece of work: they want to figure out how to embed a huge amount of domain specific knowledge into the agentic architecture.

Lev: So it's about taking all that static information—the complex physics rules and experimental procedures—and turning it into executable logic, something that actually mimics how a human expert makes strategic choices on the fly.

Kai: I’m thinking about what this means for us in the hardware side: if we can get better tool calling, maybe we can start building specialized interfaces or models specifically tuned for those physics tools.

Mira: That would be huge because it moves us away from just general intelligence toward something that understands the specific language of experimental data analysis.

Lev: From my point of view, if the AI could generate perfectly compliant code blocks, running that on real hardware becomes much more reliable and less prone to catastrophic failure during complex jobs.

Kai: So the goal isn't just a smarter reasoning engine; it’s building a system where the execution layer is just as precise as the decision-making layer.

Mira: Exactly, and they suggest focusing on how to represent those expert heuristics so that when DrSai decides what to do, it’s making choices that actually look like a seasoned physicist.

Lev: It sounds like the next step is less about increasing the reasoning depth and more about fixing the precision of execution and deepening that domain integration.

Kai: That leads us nicely into how this architecture might be tested further, looking at those specific performance metrics they used, like success rates under different retry budgets.

Mira: We need to look closely at those limitations mentioned—what the current system can’t do—to see if those proposed improvements actually fix the core problems they identified.

Conclusion: Kai: So, we've covered how DrSai uses a multiagent system to automate analysis workflows at BESIII, showing it can handle complex physical consistency across different decay channels like J psi decays and estimate systematic uncertainties.

Mira: Right, and the core idea is that this AI doesn't just follow instructions; it uses a structured decomposition of tasks to manage the entire experimental process from an initial physics goal down to preliminary data fitting.

Lev: From my side, it’s interesting that they found a performance bottleneck in functional calling capability rather than just reasoning depth across all the tested large language models.

Kai: So, if we look at what this paper suggests for the future, it really shifts the focus toward improving how the AI actually interacts with external tools and generating precise code.

Mira: They’re pushing for specialized synthesis layers or fine-tuned models that have a syntax level mastery of professional physics tools, which would make it much better at writing executable sequences.

Lev: That seems like a solid direction because if you can get the AI to write accurate code blocks based on those configuration files, the overhead lands much more predictably on real hardware.

Kai: I think the biggest long term challenge they’ve laid out is getting that domain specific knowledge embedded so the AI can make strategic decisions that look like an expert physicist's judgment.

Mira: It’s about taking all that complex static information and making it into logic, not just following a script blindly.

Lev: So the next hurdle isn't just more reasoning power; it’s building a proper way to transform expert thought into machine executable code that mirrors actual decision-making.

Kai: It’s a big step toward having these agents act as competent partners in the field, rather than just assistants for asking questions.

Mira: Yes, and seeing this validated at BESIII gives us real confidence that this agentic approach can handle the rigor of real-world experimental frameworks.

Lev: We’ll keep watching how they tackle those limitations on functional calling because that precision is what separates a good assistant from an actual tool for heavy lifting.

More episodes

← Home