Uncertainty Quantification for LLM Function-Calling

summary

The gist

Large Language Models (LLMs) are increasingly deployed to autonomously solve real-world tasks, and this work presents what is, to our knowledge…

In short

This work evaluates Uncertainty Quantification (UQ) methods for LLM Function-Calling (FC), a task where models generate structured code/calls. It benchmarks various UQ techniques against a specific dataset, finding that adapting methods to FC output characteristics—like using Abstract Syntax Trees or selecting semantically meaningful tokens—improves performance over standard single-sample and multi-sample approaches.

Key concepts

Function-Calling (FC)
This is when a Large Language Model (LLM) is asked to generate specific code or calls to external functions, rather than just answering a question. It requires the model to output structured data that represents an action, like calling a specific API function with correct parameters.
Uncertainty Quantification (UQ)
UQ methods are techniques used to measure how confident an LLM is in its own prediction. In this context, they estimate the probability that the generated function call is correct or incorrect, helping users decide whether to trust the model's output.
Semantically Meaningful Tokens (SMT)
SMT refers to specific words or tokens within an LLM's output that are crucial for determining correctness. These are tokens where the model must choose between options that lead to a right or wrong function call, such as deciding which function name to use.
Abstract Syntax Trees (AST)
ASTs are a tree-like structure used in programming to represent the structure of code. When applied here, ASTs help cluster different function calls based on the actual structure of their arguments, allowing multi-sample UQ methods to better understand semantic similarity.

Terminology used across episodes

This episode discusses

The paper

Uncertainty Quantification for LLM Function-Calling · Read on arXiv

Zihuiwen Ye, Lukas Aichberger, Michael Kirchhof, Sinead Williamson, Luca Zappella, Yarin Gal

University of Oxford · Apple

Large Language Models (LLMs) are increasingly deployed to autonomously solve real-world tasks. A key ingredient for this is the LLM Function-Calling paradigm, a widely used approach for equipping LLMs with tool-use capabilities. However, an LLM calling functions incorrectly can have severe implications, especially when their effects are irreversible, e.g., transferring money or deleting data. Hence, it is of paramount importance to consider the LLM's confidence that a function call solves the task correctly prior to executing it. Uncertainty Quantification (UQ) methods can be used to quantify this confidence and prevent potentially incorrect function calls. In this work, we present what is, to our knowledge, the first evaluation of UQ methods for LLM Function-Calling (FC). While multi-sample UQ methods, such as Semantic Entropy, show strong performance for natural language Q&A tasks, we find that in the FC setting the entropy-based multi-sample methods we evaluate offer no clear advantage over simple single-sample UQ methods. Additionally, we find that the particularities of FC outputs can be leveraged to improve the performance of existing UQ methods in this setting. Specifically, multi-sample UQ methods benefit from clustering FC outputs based on their abstract syntax tree parsing, while for single-sample UQ methods, restricting the logit-based score to semantically meaningful tokens leaves discrimination essentially unchanged but improves calibration and, by smaller margins, the utility of the score under abstention.

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "Uncertainty Quantification for LLM Function-Calling".

Jane: Here's the gist of the paper. Large Language Models (LLMs) are increasingly deployed to autonomously solve real-world tasks, and this work presents what is, to our knowledge, the first evaluation of Uncertainty Quantification (UQ) methods for LLM Function-Calling (FC) <ref:2604.22985#pg2> While multi-sample UQ methods show strong performance for natural language Q&A tasks, they offer no clear advantage over simple single-sample UQ methods in the FC setting <ref:2604.22985#pg4> The particularities of FC outputs can be leveraged to improve existing UQ methods by tailoring them to the specific characteristics of these outputs <ref:2604.22985#pg2> Benchmarking and Methodology The authors build the first benchmark for UQ methods in Function-Calling settings based on the Berkeley Function Calling Leaderboard dataset (BFCL) <ref:2604.22985#pg4> This benchmark evaluates various UQ approaches by assessing how well their confidence estimates predict the correctness of function calls outputted by LLMs <ref:2604.22985#pg4> They reimplement several UQ methods, ranging from simple single-sample approaches based on model logits to more sophisticated multi-sample approaches that incorporate semantic equivalence between samples <ref:2604.22985#pg4> The evaluation is quantified using the Area Under the Receiver Operating Characteristic (AUROC) metric, which measures how well uncertainty estimates discriminate correct from incorrect function calls <ref:2604.22985#pg4> Performance Comparison The findings indicate that in the FC setting, unlike in natural language Q&A settings, more involved multi-sample methods do not outperform simple single-sample UQ methods <ref:2604.22985#pg4> Specifically, Semantic Entropy (SE) does not outperform Greedy Negative Log-Likelihood (G-NLL) as the best single-sample method <ref:2604.22985#pg4> The authors then investigate improving existing UQ methods by adapting them to FC output characteristics <ref:2604.22985#pg2> They adapt the clustering strategy in Semantic Entropy by using Abstract Syntax Trees (AST) to cluster FC outputs, and they adjust G-NLL and other single-sample UQ methods by selecting only semantically meaningful tokens when calculating UQ scores <ref:2604.22985#pg2> Adaptation of Single-Sample Methods The authors define Semantically Meaningful Tokens (SMT) as those corresponding to positions where the LLM has to decide between tokens that likely lead to either correct or incorrect outputs <ref:2604.22985#pg4> They define these tokens based on analyzing FC outputs, identifying positions related to deciding if functions are called, if the right function is called, if the right parameter is used, and if the right parameter value is chosen <ref:2604.22985#pg4> Implementing SMT Extraction involves a rule-based approach that classifies tokens based on their role in forming a correct function call output <ref:2604.22985#pg4> This leads to an adaptation of single-sample methods referred to as Semantically Meaningful Tokens (SMT) <ref:2604.22985#pg4> Overall, the work demonstrates that one can leverage FC output particularities to improve over existing UQ methods for LLMs, both for single- and for multi-sample UQ methods <ref:2604.22985#pg4> Specifically, they adapt the semantic-entailment method of multi-sample UQ methods by using Abstract Syntax Trees <ref:2604.22985#pg2> For single-sample UQ methods, they use only semantically meaningful tokens when calculating their UQ scores <ref:2604.22985#pg4> Limitations include the difficulty in comparing results on frontier models and the need for analysis in more challenging settings like multi-turn or agent settings <ref:2604.22985#pg4> The computational cost of multi-sample methods remains a barrier to surpassing single-sample methods <ref:2604.22985#pg4> Compute Infrastructure LLM inference for the BFCL tasks was performed using vLLM on a single node with 4 A100 GPUs, taking roughly 40 minutes in total to generate the greedy samples across all task datasets and models <ref:2604.22985#pg4> Generating 10 high-temperature samples per request required 3 hours in total for all task datasets and models <ref:2604.22985#pg4> Computation of all UQ scores for any single model and any (individual or combined) task dataset took less than a minute on a 2021 Apple MacBook Pro with M1 Max chip and 64GB RAM <ref:2604.22985#pg4> Risk Coverage Analysis The risk-coverage curves illustrate what the AUROC values correspond to in terms of potential accuracy gains thanks to UQ-based abstention <ref:2604.22985#pg4> For example, not executing a function call for the 30% of requests with highest G-NLLSMT UQ scores increases accuracies from < 92% to 96% for Simple tasks and from 75% to 85% for All Combined tasks <ref:2604.22985#pg4> The gains are similar across task combinations, reaching approximately +10% once the irrelevance task is added <ref:2604.22985#pg4> Calibration Insights The smooth Expected Calibration Error (smoothECE) shows that AVG achieves the worst calibration among all methods, being largely overconfident with confidence values higher than the accuracy <ref:2604.22985#pg4> Ptrue and G-NLL are underconfident, while MAX achieves the best calibration <ref:2604.22985#pg4> This illustrates that ECE does not necessarily correlate with AUROC <ref:2604.22985#pg4> All multi-sample entropy-based methods do not output probabilities, so their calibration cannot be evaluated directly AST Decoding Errors and Effective Sample Sizes On the All Combined split, 3.4% of the outputs are AST decoding errors averaged across 8 models <ref:2604.22985#pg4> The Spearman rank correlation between rankings when including AST decoding errors versus excluding them is 0.94, indicating that non-decodable function calls do not pose a risk from the perspective of preventing potentially harmful function calls <ref:2604.22985#pg4> Table 5 shows the effective sample size per model on the All Combined split Sensitivity to Hyperparameters Table 7 shows that increasing J to 30 can improve AUROC by around 0.02–0.

Title and authors: Tom: So we're looking at this paper called "Uncertainty Quantification for LLM Function-Calling". It tackles a big problem because LLMs are increasingly doing things on their own, and when they use tools or call functions—which is function calling—we need to know if they are right before they actually execute something.

Jane: Right. So, the core idea here is that these function calls can have serious consequences, like messing up data or losing money if the LLM calls a function incorrectly, and this paper presents what they call the first evaluation of uncertainty quantification methods for function calling.

Lu: Exactly. They are looking at how to quantify the model's confidence in a tool-use decision so we can stop it from making mistakes before it happens, which is critical when these actions can be irreversible <ref:2604.22985#pg1>.

Meng: From a production standpoint, that makes sense because if an LLM decides to delete something based on a faulty function call, you need a way to check that confidence score first <ref:2604.22985#pg3>.

Lalam: I’ve reviewed the summary of "Uncertainty Quantification for LLM Function-Calling", and it boils down to this: they look at how different uncertainty methods perform when the LLM is trying to call a function, and they found that while some multi-sample methods do well for normal question answering, they don't show an obvious advantage over simpler single-sample methods in this specific function calling setting <ref:2604.22985#pg4>.

Tom: So the paper benchmarks several approaches—ranging from simple methods based on the model's raw scores to more complex multi-sample methods that look at how similar different samples are semantically <ref:2604.22985#pg4>.

Jane: That's interesting because the authors suggest that they can actually improve these existing uncertainty methods by tailoring them specifically to what function calls look like, which is a key point they make in page two <ref:2604.22985#pg2>.

Lu: They explore adapting things, and for example, they take the clustering strategy used in Semantic Entropy and change it to use Abstract Syntax Trees or ASTs to group the function calls together <ref:2604.22985#pg3>.

Meng: AST matching sounds like a way to handle the structure of the function call better than just comparing text strings, which is what I'd expect in structured output like this <ref:2604.22985#pg3>.

Lalam: And for the single-sample methods, they adapt them by only looking at tokens that are semantically meaningful when calculating the uncertainty scores <ref:2604.22985#pg4>.

Tom: So this means they're not just throwing more complexity at it; they are trying to make the existing tools smarter by focusing their attention on the parts of the call that actually matter for correctness, like whether a specific function is chosen or if a parameter is correct <ref:2604.22985#pg4>.

Title and authors: Jane: That leads into how they define these semantically meaningful tokens, which they call SMTs. Basically, it's about identifying the spots in the function call where the LLM has to make a tough choice between different tokens that could lead to a right or wrong outcome <ref:2604.22985#pg4>.

Lu: They use a rule-based approach for this extraction, classifying tokens based on their role in forming a correct function call output, which is how they create these SMTs <ref:2604.22985#pg4>.

Meng: Rule-based token classification sounds like it adds some complexity to the pipeline, but if it isolates the high-leverage parts of the decision, it could be worth the effort <ref:2604.22985#pg4>.

Lalam: And they then apply this SMT concept to improve single-sample methods by only using those tokens when calculating their uncertainty scores <ref:2604.22985#pg4>.

Tom: So, one big thing here is that they show you can get better results for both the simple and multi-sample methods just by tweaking them to understand the specific characteristics of function calls instead of using a general approach <ref:2604.22985#pg4>.

Jane: That's the main implication, Tom. They demonstrate that because function calling has these unique structural properties, we can make existing uncertainty tools work better for this task <ref:2604.22985#pg1>.

Lu: It’s a practical step toward making uncertainty quantification more useful in real-world autonomous agents, moving beyond just accuracy metrics to true risk assessment <ref:2604.22985#pg3>.

Meng: I worry about the computational cost of these new methods. If we have to run AST analysis or complex token selection every time, it adds latency that we can't afford at scale <ref:2604.22985#pg4>.

Tom: That’s a valid point, Meng. But they did show some speed tests on the infrastructure side; for instance, generating the greedy samples across all datasets took about forty minutes on their setup with four A100 GPUs <ref:2604.22985#pg4>.

Jane: They also showed how this uncertainty translates into potential risk reduction. For example, by not executing a function call when its SMT uncertainty score is high, they saw accuracy jump from ninety-two percent to ninety-six percent on simple tasks <ref:2604.22985#pg4>.

Lu: That jump in accuracy shows that the abstention based on their UQ scores actually prevents potentially harmful function calls, which is what we need for safety <ref:2604.22985#pg3>.

Lalam: And they also looked at the calibration, showing that methods like MAX uncertainty can achieve the best calibration among all the ones they tested, meaning their confidence scores are more honest about how likely they are to be correct <ref:2604.22985#pg4>.

Tom: So, to wrap up on this paper, "Uncertainty Quantification for LLM Function-Calling", the main thing is that we can leverage the specific structure of function calls—using ASTs and SMTs—to make uncertainty quantification methods work better for tool use <ref:2604.22985#pg1>.

Jane: And they proved that adapting these techniques allows them to improve both the single-sample and multi-sample methods we already have, which is a useful direction for everyone working on this area <ref:2604.22985#pg1>.

Title and authors: Lu: The paper highlights how these adaptations, specifically using ASTs for clustering and SMTs for scoring, lead to tangible improvements in performance metrics like the AUROC <ref:2604.22985#pg4>.

Meng: But they also pointed out a limitation: comparing results across different frontier models is hard, and they need more work in settings that are more complex than just single function calls, like multi-turn conversations or agent workflows <ref:2604.22985#pg4>.

Tom: And one of the other things they noted is that the computational cost of multi-sample methods still makes them slower than the simpler single-sample approaches, unless you use these new adaptations <ref:2604.22985#pg4>.

Jane: So, while there are clear improvements in accuracy and calibration when we tailor the methods to function calling outputs, we still have to deal with the challenge of making sure these methods scale efficiently for real deployment <ref:2604.22985#pg4>.

Lu: It’s a solid foundation because it shows a path forward for uncertainty quantification in this specific area, moving from general Q andA performance to specialized, safer function calling decisions <ref:2604.22985#pg3>.

Meng: I'll keep an eye on the computational cost aspect. If we can deploy these adapted SMT methods without a massive overhead, that’s where this research has real value for production systems <ref:2604.22985#pg4>.

Tom: So, in summary, the work on "Uncertainty Quantification for LLM Function-Calling" shows we can use the specific details of function calls—like their syntax and argument selection—to refine how we measure model confidence <ref:2604.22985#pg1>.

Jane: It’s about moving past just seeing if a model is right or wrong in general, and instead learning how to predict which specific action it might take incorrectly <ref:2604.22985#pg1>.

Lu: This work gives us concrete methods—using AST clustering and SMT tokens—to make multi-sample UQ methods more effective for function calling tasks <ref:2604.22985#pg3>.

Meng: The practical implication is that if we can get the cost down, these tailored uncertainty scores give us a much safer way to deploy LLMs that use tools <ref:2604.22985#pg4>.

Tom: We’ll keep an eye on the future work they mention, especially how these methods fare when we move into more complex agent settings or multi-turn scenarios <ref:2604.22985#pg4>.

Jane: That sounds like a natural next step for any system that’s actually going to be used autonomously in the real world <ref:2604.22985#pg3>.

Lu: It really sets the stage for understanding how robust these systems are when they are operating under pressure, which is where we need to focus our research next <ref:2604.22985#pg4>.

Meng: Yeah, so the paper on "Uncertainty Quantification for LLM Function-Calling" provides a clear framework for making tool use safer by focusing on the structure of those calls <ref:2604.22985#pg1>.

Tom: It’s a solid piece of work because it shows that even in this complex area, we can get more reliable confidence estimates by adapting the methods to the task at hand <ref:2604.22985#pg4>.

The paper's summary: Tom: So, we're talking about how these papers—and this one specifically—are looking at uncertainty quantification for function calling, and it boils down to this: they're checking if the ways we measure a model’s confidence when it tries to call a tool are actually good enough for real use.

Jane: Exactly. Basically, the authors look at how different methods work when an AI is trying to use a function, and they found that while some complex methods do well on regular question answering, they don't show any clear advantage over simpler single-sample methods in this specific function calling setting.

Lu: Right. They’re saying you don't always need the most complicated multi-sample methods when dealing with these structured outputs from an LLM.

Meng: So what does that mean for us shipping models? If a more complex setup isn't better, we can probably stick to simpler, faster uncertainty checks for deployment.

Tom: Actually, they go further than just saying one is better than another. They show you can actually improve these existing methods by tailoring them specifically to the way function calls look.

Jane: That’s the interesting part because they propose two main ways to do that. First, they adapt the clustering strategy in a method called Semantic Entropy by using Abstract Syntax Trees or ASTs to group those function calls together instead of just matching strings.

Lu: Using ASTs means you're looking at the actual structure of the code being generated, treating it like a dictionary where arguments are interchangeable, which is much better for structured outputs than simple string matching.

Tom: And for the single-sample methods, they adapt those by only calculating uncertainty scores based on semantically meaningful tokens. They call these Semantically Meaningful Tokens or SMTs.

Jane: What does that mean practically? It means instead of looking at every single word the LLM spits out, they focus only on the parts where the model has to make a critical decision—like which function to pick, or which parameter value is right.

Lu: They define these SMTs using a rule-based approach that classifies tokens based on their role in forming a correct function call output, so it's not just guesswork.

Meng: That sounds like it adds complexity to the pre-processing step, but if it isolates the high-leverage parts of the decision, that could be worth the effort for safety.

Tom: They test this out and show that using these adaptations—like AST clustering for multi-sample methods and focusing on SMTs for single-sample methods—actually leads to better performance scores, like AUROC.

Jane: And they also looked at the results in terms of risk coverage. For example, by not executing a function call when its SMT uncertainty score is high, they saw accuracy jump from ninety-two percent to ninety-six percent on simple tasks.

Lu: That jump shows that this abstention based on their specific uncertainty scores actually prevents potentially harmful function calls in practice.

Tom: They also checked the calibration, and they found that some methods, like MAX uncertainty, achieve the best calibration among all the ones they tested. This means their confidence scores are more honest about how likely they are to be correct.

Jane: It’s a big deal because it shows that we can move past just checking if a model is right or wrong in general, and instead learn how to predict which specific action it might take incorrectly, with more accurate confidence scores.

Lu: But they did flag some limitations. They mentioned that comparing the results across different frontier models is really hard and they need to do more work in settings that are way more complex than just a single function call, like multi-turn conversations or full agent workflows.

Meng: That makes sense for me because running these AST analyses on every single request could add a lot of latency at scale.

Tom: Yeah, the computational cost of those multi-sample methods is still a barrier to surpassing the simpler single-sample approaches unless you use these new adaptations, so that's where we need to focus our engineering efforts.

Jane: So, this paper gives us a clear path forward for uncertainty quantification in this specific area by focusing on the structure of function calls and how we measure model confidence more accurately.

The paper's improvements: Tom: So, we're looking at how they actually suggested making these uncertainty quantification methods better in the paper, and it’s really about taking those ideas and putting them into a working system for function calling.

Jane: Right. The core of their suggestion is to make things adaptive. They want the system to dynamically choose between different tools, not just stick to one fixed method no matter what.

Lu: Specifically, they suggest adapting single-sample methods like Greedy Negative Log-Likelihood, or G-NLL, by making them switch over to tailored single-sample methods like Semantically Meaningful Tokens or SMTs when the model is making a call.

Meng: So instead of running the same uncertainty check every time, we could analyze the function call output first to see if it’s complex enough to warrant using a more detailed SMT analysis.

Tom: Exactly, and they back this up by showing that adapting methods like Semantic Entropy also works well when you use Abstract Syntax Trees for clustering those samples together.

Jane: That’s interesting because it means the system gets smarter about what makes a function call unique, rather than just treating every output the same way.

Lu: The implication there is that you can use these techniques to improve both single-sample and multi-sample uncertainty methods without necessarily adding massive computational overhead for every task.

Tom: They also pointed out that when they used the MAX uncertainty method alongside those SMT adaptations, it achieved the best calibration among all the methods they tested, which is a nice bonus because it means the confidence scores are more accurate.

Jane: That’s important because when you’re actually building something that uses these AI tools, you need to trust that confidence score. If MAX is better calibrated, then we can make more reliable decisions about whether to run a function or not.

Lu: It moves the goal from just getting a high AUROC score on a test set to having an uncertainty estimate that actually reflects the true probability of correctness for the specific action being taken.

Meng: I still have my reservations about deployment cost, though. If this adaptive switching requires constant deep analysis of every output, we need to make sure the performance gain is worth that added latency when it’s running live.

Tom: They did touch on that too, but they showed that for the single-sample adaptation, using only semantically meaningful tokens keeps the calculation focused and efficient.

Jane: So what this changes for someone just listening? It means we can expect future AI systems to be better at telling us *why* they are unsure about a tool call, not just *how* unsure they are.

Lu: That’s the big picture: it’s about giving these autonomous agents a way to communicate their own internal decision-making uncertainty in a way that is actually grounded in the structure of the task.

Conclusion: Tom: So we're wrapping up this look at "Uncertainty Quantification for LLM Function-Calling", and the main point is that we can take existing tools and make them much smarter by focusing on how function calls are actually structured.

Jane: Right, so it’s not just about building a bigger model; it’s about learning the specific language of tool use so we can measure risk more reliably.

Lu: The paper showed that by using Abstract Syntax Trees for clustering and Semantically Meaningful Tokens for single-sample scoring, we get concrete improvements on metrics like AUROC.

Meng: I still have my reservations about deployment cost though. If this adaptive switching requires constant deep analysis of every output, we need to make sure the performance gain is worth that added latency when it’s running live.

Tom: Yeah, the computational cost of those multi-sample methods is still a barrier to surpassing the simpler single-sample approaches unless you use these new adaptations.

Jane: And they did flag that comparing results across different frontier models is really hard, and they need to do more work in settings that are way more complex than just a single function call, like multi-turn conversations or full agent workflows.

Lu: That’s the next big challenge, moving from static function calls to dynamic agent actions where the structure changes constantly.

Lalam: I’m confident that these adaptations provide a solid foundation for making tool use safer by focusing on the syntax and parameter selection of those calls.

Tom: So in summary, "Uncertainty Quantification for LLM Function-Calling" gives us concrete methods—using ASTs and SMTs—to make multi-sample UQ methods more effective for function calling tasks.

Jane: It’s about moving past just seeing if a model is right or wrong in general, and instead learning how to predict which specific action it might take incorrectly, with more accurate confidence scores.

Lu: This work gives us a clear path forward for uncertainty quantification in this specific area by focusing on the structure of function calls and how we measure model confidence more accurately.

Meng: I'll keep an eye on the future work they mention, especially how these methods fare when we move into more complex agent settings or multi-turn scenarios.

Tom: So that’s it for this paper. It shows that even in this complex area, we can get more reliable confidence estimates by adapting the methods to the task at hand.

Jane: It’s about moving past just seeing if a model is right or wrong in general, and instead learning how to predict which specific action it might take incorrectly.

Lu: The implication there is that these tailored uncertainty scores provide a more robust way to assess risk when an AI is using tools.

More episodes

← Home