Towards More Trustworthy and Interpretable LLMs for Code through Syntax-Grounded Explanations
cs.SE, cs.AI, cs.LG
Submitted: 2024-07-12
Updated: 2026-09-26
DOI: 10.1145/3771282
Code: https://github.com/numpy/numpy
Project page: https://christophm.github.io/interpretable-ml-book
License: http://creativecommons.org/licenses/by-nc-nd/4.0/
The gist: Trustworthiness and interpretability are inextricably linked concepts for LLMs.
Terminology
Abstract
Trustworthiness and interpretability are inextricably linked concepts for LLMs. The more interpretable an LLM is, the more trustworthy it becomes. However, current techniques for interpreting LLMs when applied to code-related tasks largely focus on accuracy measurements, measures of how models react to change, or individual task performance instead of the fine-grained explanations needed at prediction time for greater interpretability, and hence trust. To improve upon this status quo, this paper introduces ASTrust, an interpretability method for LLMs of code that generates explanations grounded in the relationship between model confidence and syntactic structures of programming languages. ASTrust explains generated code in the context of syntax categories based on Abstract Syntax Trees and aids practitioners in understanding model predictions at both local (individual code snippets) and global (larger datasets of code) levels. By distributing and assigning model confidence scores to well-known syntactic structures that exist within ASTs, our approach moves beyond prior techniques that perform token-level confidence mapping by offering a view of model confidence that directly aligns with programming language concepts with which developers are familiar. To put ASTrust into practice, we developed an automated visualization that illustrates the aggregated model confidence scores superimposed on sequence, heat-map, and graph-based visuals of syntactic structures from ASTs. We examine both the practical benefit that ASTrust can provide through a data science study on 12 popular LLMs on a curated set of GitHub repos and the usefulness of ASTrust through a human study.
Sources
- Unified Pre-training for Program Understanding and Generation
- Sampling in Software Engineering Research: A Critical Review and Guidelines
- Evaluating Large Language Models Trained on Code
- An Empirical Study on the Usage of BERT Models for Code Completion
- Towards A Rigorous Science of Interpretable Machine Learning
- Comparing Explanation Methods for Traditional Machine Learning Models Part 1: An Overview of Current Methods and Quantifying Their Disagreement
- Towards Automatic Concept-based Explanations
- CodeSearchNet Challenge: Evaluating the State of Semantic Code Search
- Visualizing and Understanding Recurrent Networks
- On the Reliability and Explainability of Language Models for Program Generation
- Trustworthy and Synergistic Artificial Intelligence for Software Engineering: Vision and Roadmaps
- AST-Probe: Recovering abstract syntax trees from hidden representations of pre-trained language models
- Interpretable Machine Learning -- A Brief History, State-of-the-Art and Challenges
- Beyond Word Importance: Contextual Decomposition to Extract Interactions from LSTMs
- CodeBLEU: a Method for Automatic Evaluation of Code Synthesis
- Probing Pretrained Models of Source Code
- What Do They Capture? -- A Structural Analysis of Pre-Trained Language Models for Source Code
- A Systematic Literature Review on the Use of Deep Learning in Software Engineering Research
- A Systematic Evaluation of Large Language Models of Code
Related papers
- Falsification-Based Verification of LLM-Generated Optimization Models: Sound Test Batteries and Their Detection Limits
- GitSkills: A Dataset of Agent Skills on GitHub
- SABER: Benchmarking Operational Safety of LLM Coding Agents in Stateful Project Workspaces
- PackMonitor: Enabling Zero Package Hallucinations Through Decoding-Time Monitoring
- IntentCoding: Amplifying User Intent in Code Generation
- Incentives and Outcomes in Bug Bounties