BankerToolBench: Evaluating AI Agents in End-to-End Investment Banking Workflows

arXiv:2604.11304 · cs.AI · Submitted 2026-04-13 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: Today's paper: "BankerToolBench: Evaluating AI Agents in End-to-End Investment Banking Workflows".

Jane: The combination of these excerpts allows for a comprehensive, high-fidelity understanding of this work's scope, methodology, contributions, and critical findings.

Tom: First, who's behind it and why it matters.

Title and authors: Tom: So we're looking at a paper called BankerToolBench: Evaluating AI Agents in End-to-End Investment Banking Workflows. Jane, what's the main takeaway from the title itself?

Jane: It sounds pretty specific, doesn't it? It’s not just about making an AI write a report; it’s about testing if an AI can actually handle the whole process of doing actual investment banking work.

Tom: Exactly. The authors, Handshake AI and their team, are tackling the problem that existing benchmarks just don't cover the real professional stuff. They say they collaborated with five hundred two investment bankers from leading firms to build this thing <ref:2604.11304#pg1,collaborated with 502 investment bankers from leading firms>.

Lu: So it’s built on real-world work environments rather than just textbook problems. That’s a big difference for testing agent capability, right?

Meng: I wonder how much of that real-world context is actually baked into the environment, or if the agents are just given a bunch of documents to look at.

Lalam: It seems they really focused on making sure the AI has access to actual tools and data rooms, not just free internet search.

The paper's summary: Jane: Okay, so what is BankerToolBench actually doing in detail? They are setting up a reinforcement learning environment where an agent gets a prompt from a banker, uses tools like market data APIs and SEC EDGAR APIs to navigate data rooms.

Tom: And the goal is for that agent to generate deliverables—things like Excel spreadsheets, PowerPoint decks, and PDF reports—all in one go. The whole point is testing if an AI can handle tasks that could take a human up to twenty-one hours.

Lu: They structured the tasks with a specific grading rubric from senior bankers, which means the evaluation isn't just about whether the answer is technically correct, but how well it follows professional standards.

Meng: That sounds intense. So it’s not just checking if the numbers add up; they’re checking if the entire package—the spreadsheet and the slide deck—makes sense together.

Lalam: The paper notes that approximately thirty-six percent of these tasks require producing a complete package, which is a significant part of their focus.

The paper's improvements: Tom: Now, the authors point out that just having the benchmark isn't enough; they are proposing specific improvements to make it better for testing AI agents. They are focusing on things like making sure the AI can handle complex financial modeling tasks, like convertible structures or IPO pricing.

Jane: And they highlight that a major part of their contribution is how they designed their verifier, which is an agent called Gandalf four. This verifier acts as an agent-as-a-judge model to grade the deliverables against over one hundred detailed criteria.

Lu: The key improvement here is moving beyond simple text evaluation to checking the actual output files, like that Excel workbook, and using this verifier to ensure accuracy across all those components.

Meng: From a deployment standpoint, that sounds like a lot of work for the verification system itself. So they’re trying to build something robust enough that it can actually grade at scale without breaking down on complex financial formulas.

Lalam: They emphasize that the resulting Excel workbook needs to be accurate and auditable, and even minor analytical or presentation errors are unacceptable, which really raises the bar for what an AI has to achieve.

Conclusion: Jane: So, to wrap up this discussion on BankerToolBench: it’s a rigorous test of end-to-end workflow execution in investment banking using real industry data and human rubrics. The main implication is that we need benchmarks that test the entire chain from request to final deliverable.

Tom: Right. And the results are pretty sobering for frontier models like GPT-five point four; they only manage to pass about sixteen percent of these complex tasks autonomously, which suggests they aren't ready for real banking work yet.

Lu: The paper finds that the failure isn't usually one small thing; it’s systemic breakdowns in cross-artifact consistency and unreliable handling of assumptions, which shows models struggle with maintaining coherence across many steps.

Meng: For us shipping things to production, that means we can’t just look at raw benchmark scores; we have to see if the model can actually be trusted with the whole workflow because isolated parts are not enough.

Lalam: The paper notes specific failure modes, like code and formula generation being a big weak spot, which tells us exactly where the AI is failing in these financial tasks.

Jane: It’s clear that the critical metric isn't whether an AI can do one small part of a task well, but whether it can be trusted with the whole workflow when you give it a real-world request like this.

Tom: Exactly. So we see this as a massive step toward understanding what’s actually required for AI agents to handle high-value, labor-intensive professions like finance.

Lu: It’s a good foundation for future work because it shows the direction we need to go to build systems that integrate capability into stable execution.

Meng: We'll keep watching these results and seeing if we can get those models to pass that reliability threshold when they are actually deployed on real traffic.

Lalam: So, BankerToolBench is a really high-fidelity attempt at evaluating AI agents in an end-to-end fashion for investment banking workflows.

McGill University

cs.AI

Submitted: 2026-04-13

Updated: 2026-04-13

Journal ref: NeurIPS 2026

Code: https://github.com/Handshake-AI-Research/bankertoolbench

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 90/100

The gist: The combination of these excerpts allows for a comprehensive, high-fidelity understanding of this work's scope, methodology, contributions, and critical findings.

Key concepts

BankerToolBench (BTB): BTB is a benchmark designed to evaluate if AI agents can reliably perform the full cycle of junior investment banking tasks. It simulates real work by requiring agents to use industry tools, navigate data rooms, and generate complex outputs like financial models and pitch decks.
End-to-End Workflow Evaluation: This method tests an agent's ability to complete a multi-step professional task from start to finish. Instead of testing single capabilities, BTB assesses the agent's success in navigating data, using tools, reasoning through complex financial models, and producing final deliverables.
Cross-Artifact Consistency: This refers to the failure of an AI agent to maintain logical coherence across different parts of a single output package. For example, the formulas in an Excel sheet might not match the assumptions made in the accompanying PowerPoint presentation, leading to systemic errors.
Agent-as-a-Judge (Gandalf 4): This is an automated grading system used within BTB that acts as a sophisticated AI judge for the deliverables. It scores outputs against over 100 detailed rubric criteria, aiming to mimic the judgment of veteran investment bankers.

Terminology

Benchmark — 3× in this paper · explained in 2 episodes
AI Agents — 2× in this paper · explained in 2 episodes
Agentic AI — 1× in this paper · explained in 1 episode
Consistency — 1× in this paper · explained in 3 episodes
Fidelity — 1× in this paper · explained in 2 episodes

Summary

The combination of these excerpts allows for a comprehensive, high-fidelity understanding of this work's scope, methodology, contributions, and critical findings.

Here is the detailed synthesis:

Comprehensive Research Summary: BankerToolBench (BTB)

BankerToolBench (BTB) is a novel open-source benchmark meticulously designed to evaluate the end-to-end analytical workflows routinely performed by junior investment bankers. The core premise of BTB is to test whether Large Language Models (LLMs) and autonomous agents can reliably execute complex, multi-step professional tasks that span data navigation, tool utilization, and the generation of multi-file deliverables—a process that can take human bankers up to 21 hours.

Methodology and Framework:

BTB is structured as a reinforcement learning environment (RLE) within the Harbor framework. It provides agents with a rich operational context by pre-installing necessary software and granting access to critical industry tools, such as a market data platform API and an SEC EDGAR API. Tasks are composed of three essential elements: a natural language prompt, access to a specific data room (containing relevant documents and tools), and an expert-authored rubric comprising over 100 detailed grading criteria.

Scope of Evaluation:

The benchmark is designed for high fidelity, focusing on economically meaningful professional workflows. The tasks are highly diverse, spanning several core investment banking functions:

Financial Modeling: This includes complex tasks such as Use of Proceeds, Convertible Structures, Operating Models, and IPO Pricing. It also covers valuation techniques like DCF, Trading Comps, Price Ranges, and Sum-of-the-Parts analysis.
Industry Coverage: The benchmark ensures broad applicability by covering all GICS sectors. Crucially, it mandates sector-specific metrics and standards; for instance, the Technology sector requires specific metrics like revenue multiples (EV/Revenue, EV/ARR for SaaS), rule of 40, and net revenue retention (NRR). Media and Telecom tasks focus on subscriber growth and ARPU, while Financial Institutions require estimates for regulatory capital or net interest margin (NIM).
Deliverable Complexity: A significant feature is the requirement for multi-file outputs. Approximately 36% of tasks necessitate producing a complete package including Excel spreadsheets, PowerPoint pitch decks, and PDF/Word reports within a single deliverable submission. The most common output format is the Excel spreadsheet (86% of tasks), which demands sophisticated, formula-driven linkages across multiple tabs for sensitivity tables and scenario analyses.

Evaluation Rigor (The Verifier):

To move beyond simple text-based evaluation, BTB incorporates an agentic verifier named Gandalf 4. This agent acts as an agent-as-a-judge model, grading complex deliverable files against the 100+ rubric criteria. The verifier demonstrates high reliability, achieving a Cohen’s kappa of 0.756 with consensus, placing its performance squarely within the range of human-human agreement (epsilon = 0.692–0.815). This suggests the automated grading mechanism operates at a level comparable to a trained human grader.

Key Findings and Critical Insights:

The empirical evaluation reveals a critical threshold for deployability: current frontier models, including GPT-5.4, do not yet meet the reliability standard required for delegating real banking workflows.

Failure of Coherence: The dominant failure modes observed are not isolated capability shortcomings but systemic breakdowns in end-to-end execution. These include severe issues with cross-artifact consistency, unreliable handling of assumptions and judgment calls, and the propagation of small errors into system-wide invalidity. This strongly suggests that models struggle to maintain coherence across extended chains of interdependent sub-tasks.
Performance Gap: Despite leading in raw benchmark scores, GPT-5.4 only manages to pass approximately 16% of tasks, indicating it is unable to complete the majority of complex investment banking workflows autonomously.
Decision-Relevant Threshold: The study concludes that the critical metric for success is not whether a model can complete isolated parts of a task, but whether it can be trusted with the whole workflow. Benchmark gains that do not translate into this level of reliability risk overstating real-world readiness and potentially misguiding deployment decisions.
Failure Mode Breakdown: The specific areas where models falter most significantly are:

Code & Formula Generation (41%)
Reasoning & Logic (27%)
Retrieval & Persistence (18%)
Grounding & Fabrication (13%)

Mitigation and Future Directions:

The research identifies that isolated capability improvements are insufficient; systems must integrate these capabilities into stable, end-to-end execution. Furthermore, incorporating banker-specific domain and formatting context directly into prompts significantly enhances performance. Post-training models have shown substantial gains—up to 13× for smaller models on BTB tasks when using rubric scores as the reward signal.

Limitations:

The paper acknowledges limitations, including its reliance on standardized data sources rather than live deal data, the evaluation of discrete workflows rather than full social context, and a US-centric design. Nevertheless, BTB is positioned as one of the first high-fidelity attempts to rigorously evaluate AI agents on economically meaningful professional workflows in an end-to-end fashion.

Improvements for AI systems

Bold header: End-to-End Workflow Execution Capability

The improved system can execute end-to-end analytical workflows by integrating tool calling, information retrieval, analytical reasoning, file manipulation, information organization, accuracy verification, presentation polish across multiple files and tools.

Bold header: Multi-File Deliverable Coherence

The system will generate deliverables that achieve cross-artifact reconciliation, addressing the core failure mode where agents struggle with breakdowns in cross-artifact consistency and ensuring names, terminology, numbers, and formulas properly match across files.

Bold header: Robust Financial Model Integrity

The agent will ensure financial models are functional by adhering to constraints like those in the prompt template, specifically by implementing recalculation protocols when using Excel libraries such as openpyxl to handle computed values correctly.

Bold header: Client-Ready Output Quality

The system can produce outputs that meet professional standards because the evaluation relies on 100+ rubric criteria defined by veteran investment bankers, ensuring the deliverables are assessed for immediate usability and polish and client-facing work.

Bold header: Enhanced Domain Knowledge Integration

Performance is significantly improved when models are given context, as demonstrated by the finding that both domain and formatting context significantly improve performance when added to the prompts, suggesting models can be enhanced through training on IB-specific data which encodes the missing domain knowledge.