Legal Compliance Evaluation of Smart Contracts Generated By Large Language Models
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.
Tom: I'm Tom.
Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.
Tom: We're looking at the paper: "Legal Compliance Evaluation of Smart Contracts Generated By Large Language Models".
Jane: Here's the gist of the paper. Smart contracts can implement and automate parts of legal contracts, but ensuring their legal compliance remains challenging <ref:2506.00943#pg2> Given the recent advances of Large Language Models (LLMs) in code generation, we investigate their ability to generate legally compliant smart contracts directly from natural language legal contracts, addressing these challenges <ref:2506.00943#pg2>.
Tom: We propose a novel suite of metrics to quantify legal compliance based on modeling both legal and smart contracts as processes and comparing their behaviors <ref:2506.00943#pg2>.
Jane: We focus on the alignment between the behaviors captured in a natural-language legal contract and those expressed in the corresponding smart contract, using this as a proxy to assess LLMs’ capability to generate legally compliant code <ref:2506.00943#pg2>.
Tom: To facilitate this comparison, we adopt a contracts-as-processes view
12: , modeling both legal and smart contracts as Petri Nets (PNs)
13: to analyze their behaviors <ref:2506.00943#pg2>.
Paper summary: Tom: So wrapping up this discussion on "Legal Compliance Evaluation of Smart Contracts Generated By Large Language Models," the paper shows that while large language models can generate code, we still need a rigorous way to measure how well that code actually adheres to the legal requirements <ref:2506.00943#pg2>.
Jane: The main implication is that instead of just hoping the AI gets it right, we now have metrics like fitness and precision that let us quantify the gap between what’s in a legal document and what’s actually in a smart contract <ref:2506.00943#pg1>.
Lu: This is significant because it moves us away from just saying "the AI did well" to having objective data on *how* well it performed <ref:2506.00943#pg2>.
Meng: For deployment, this means we can start building systems where the compliance score from these metrics is a necessary gate before any code goes live <ref:2506.00943#pg1>.
Tom: Exactly. And the work by Chanuka Wijayakoon and her team gives us a foundation for automated and self-improving development workflows, provided we use these tools correctly <ref:2506.00943#pg2>.
Jane: It’s about providing a way to help the AI learn and improve its legal generation skills by giving it concrete feedback based on how it aligns with our defined legal processes <ref:2506.00943#pg1>.
Lu: Ultimately, this paper is about building the infrastructure for legally sound AI-generated code, which is a necessary step before we can really rely on these systems for anything substantial <ref:2506.00943#pg2>.
Conclusion: Tom: So we're wrapping up on this paper, "Legal Compliance Evaluation of Smart Contracts Generated By Large Language Models." Essentially, they've set up a way to actually check if those smart contracts the large language models write are actually following the law.
Jane: That makes sense. Before this, you could just hope the AI did a good job with legal stuff, but now they’re giving us specific ways to measure that gap between what’s in the contract and what’s in code.
Lu: The core idea here is modeling both a legal document and a smart contract as processes—using something called Petri Nets—so they can compare their behaviors directly, not just look at the text superficially.
Meng: I'm interested because it moves beyond just syntax checking. It tries to capture the actual *logic* of the legal requirements, which is where most AI fails when things get messy and complex.
Lalam: I can confirm that they introduced three main metrics: fitness, which checks how many behaviors from the legal contract are in the smart contract; precision, which looks at how much extra behavior you added that wasn't actually required; and functional equivalence score, which just checks if both documents end up in the same final state.
Tom: Those metrics sound like they’re giving us a real score instead of a vague "it probably worked" feeling. I’m curious about the results themselves.
Jane: Yeah, the experimental setup involved taking five legal contracts and asking four different large language models to generate twenty smart contracts from them. And what they found was that while every model wrote syntactically correct code, there was a big difference in how compliant they were.
Lu: They found that larger models generally showed higher compliance scores, which suggests the size or training data of the language model plays a role in understanding legal nuance better.
Meng: That's interesting—so bigger models are better at this specific task, even though I’d bet on needing way more testing on real-world traffic to know if that holds up when things break at scale.
Lalam: They did find specific examples, like for a contract called GCDC ToS, where one model hit a high score of zero point five for both fitness and functional equivalence. That’s concrete data from the comparison.
Tom: So, it sounds like the authors are showing that while LLMs are great at writing code structure, we need these kinds of process-based metrics to move toward actually trusting them with anything legally sensitive.
Jane: Exactly. It's not about replacing lawyers or developers; it’s about giving us a tool to review and refine the AI’s output much more effectively before deployment.
Lu: The implication is that we can start building automated workflows where the AI generates a first draft, gets scored by these metrics, and then only needs human experts to focus on the areas with low scores.
Meng: If they can automate that scoring process, it could drastically reduce the manual review time for generating smart contracts. That’s something I think would actually make production feasible.
Tom: So we're moving from just being amazed by what the AI writes to having a measurable, quantifiable standard for legal correctness in code generation.
Jane: Right. And that foundation is what opens up the door for future work, looking at how these metrics can be used in self-improving systems where the models learn from their compliance failures over time.
Chanuka Wijayakoon, Hai Dong, H.M.N. Dilum Bandara, Zahir Tari, Anurag Soin
School of Computing Technologies, RMIT University · CSIRO's Data61 Institute for Cyber-Physical Systems Research Centre
cs.SE, cs.AI
Submitted: 2025-06-01
Updated: 2025-06-01
Comments: Accepted for publication at IEEE International Conference on Blockchain and Cryptocurrency (ICBC) 2025
Journal ref: 2025 IEEE International Conference on Blockchain and Cryptocurrency (ICBC), Pisa, Italy, 2025, pp. 1-9
DOI: 10.1109/ICBC64466.2025.11114661
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 85/100
The gist: Smart contracts can implement and automate parts of legal contracts, but ensuring their legal compliance remains challenging <ref:2506.00943#pg2> Given the recent advances of Large Language Models…
Key concepts
- Fitness
- This metric measures how many specific behaviors or clauses from the original legal contract are actually present and strictly matched in the generated smart contract. It quantifies the degree of coverage of the required legal obligations within the resulting code.
- Precision
- Precision assesses how much extra, irrelevant behavior an LLM adds to its output beyond what is explicitly required by the legal document. High precision means less unnecessary code is generated, focusing only on relevant contract elements.
- Functional Equivalence Score (FES)
- The FES evaluates overall functional similarity by checking if both the legal contract and the smart contract result in the same legally equivalent state at their conclusion. It focuses on the ultimate outcome rather than just matching every single step in sequence.
Terminology
Summary
Smart contracts can implement and automate parts of legal contracts, but ensuring their legal compliance remains challenging <ref:2506. p. 0 soin@student. Abstract—Smart contracts can implement and automate parts
of legal contracts, but ensuring their legal compliance remains
challenging. Existing approaches such as formal specification,
verification, and model-based development require expertise in
both legal and software development domains, as well as extensive
manual effort. Given the recent advances of Large Language
Models (LLMs) in code generation, we investigate their ability
to generate legally compliant smart contracts directly from
natural language legal contracts, addressing these challenges. Given the recent advances of Large Language
Models (LLMs) in code generation, we investigate their ability
to generate legally compliant smart contracts directly from
natural language legal contracts, addressing these challenges. We
propose a novel suite of metrics to quantify legal compliance
based on modeling both legal and smart contracts as processes
and comparing their behaviors.Where the paper says it
Where the paper says it
Where the paper says it
Methodology Overview
We focus on the alignment between the behaviors captured in a natural-language legal contract and those expressed in the corresponding smart contract, using this as a proxy to assess LLMs’ capability to generate legally compliant code <ref:2506.00943#pg2>. To facilitate this comparison, we adopt a contracts-as-processes view [12], modeling both legal and smart contracts as Petri Nets (PNs) [13] to analyze their behaviors <ref:2506. p. 0 We focus on the alignment between
the behaviors captured in a natural-language legal contract
and those expressed in the corresponding smart contract, and
use it as a proxy to assess LLMs’ capability to generate
legally compliant code. To facilitate this comparison, we adopt
a contracts-as-processes view [12], modeling both legal and
smart contracts as Petri Nets (PNs) [13] to analyze their
behaviors. To facilitate this comparison, we adopt
a contracts-as-processes view [12], modeling both legal and
smart contracts as Petri Nets (PNs) [13] to analyze their
behaviors. We introduce a novel suite of metrics for legal compliance:
1) fitness, measuring how many behaviors from the legal contract are captured in the smart contract; 2) precision, assessing
how many behaviors in the smart contract are relevant to
the legal contract; and 3) functional equivalence score (FES),
providing a flexible assessment of the alignment between theWhere the paper says it
Where the paper says it
Metrics Definition
The fitness metric measures how closely the behaviors of a smart contract (SC) match those of the legal contract (LC) based on strict event equivalence <ref:2506.00943#pg2>. Fitness is defined as the ratio of behaviors that strictly match between the contracts to the total number of behaviors in the legal contract, formalized as F(SC, LC) = R ∩ Q / R <ref:2506. p. 4 Additionally, when multiple behaviors in a smart contract map
to a behavior in the legal contract, they are counted as one. The fitness F(SC, LC) of a smart contract to the legal
contract is defined as the ratio of behaviors that strictly match
between the contracts to the total number of behaviors in the
legal contract. 5,
respectively, A better captures the set of expected behaviors
from the legal contract. Precision
Precision measures how much extra behavior a smart contract introduces beyond those in the legal contract. Functional Equivalence Score (FES)
FES measures the overall functional similarity between the
legal contract and the smart contract, relaxing the strict event
equivalence requirement of the fitness metric. It compares
whether at the end of corresponding behaviors, both the legal
and smart contracts reach legally equivalent states, even if the
event sequences differ.Where the paper says it
Where the paper says it
Where the paper says it
Experimental Setup and Findings
We selected four LLMs—GPT-4o, Gemini 1.5 Pro, Claude 3. p. 0 While LLMs can produce syntactically correct code [
7], generating smart contracts from natural-language legal agreements
GPT-4o
Gemini
1. 5 Pro
Claude
3. 5 Pro
Claude
3. 5 Sonnet
Llama 3. We
propose a novel suite of metrics to quantify legal compliance
based on modeling both legal and smart contracts as processes
and comparing their behaviors. We select four LLMs, generate
20 smart contracts based on five legal contracts, and analyze
their legal compliance. We select four LLMs, generate
20 smart contracts based on five legal contracts, and analyze
their legal compliance. We find that while all LLMs generate
syntactically correct code, there is significant variance in their
legal compliance with larger models generally showing higher
levels of compliance. It is defined as the ratio of behaviors
in the smart contract that are functionally equivalent to those
in the legal contract, regardless of strict event matching:
F ES(SC, LC) = |(R ∩ Q)
∗
|
|R|
(3)
TABLE II
OVERVIEW OF LEGAL CONTRACTS
Contract Name Obligations Powers Behaviors*
GCDC Terms of Service 0 7 12
Meat sale 3 3 12
Pizza delivery 3 2 3
Shipper-Carrier 3 0 3
Transactive energy 3 3 24
* Behaviors in the reachability graph of the legal contract’s PN. TABLE III
LARGE ANGUAGE MODELS AND THEIR CHARACTERISTICS
Model Knowledge Context Max Open
Cutoff Window Output Source
Claude 3. 3) Compliance Metrics: Table IV shows the compliance
metrics for selected contracts. For GCDC, we observe:
• Claude Sonnet has the highest fitness score at 0. Without this functionality, LLM outputs do not reach the
same final state as the legal contract. • Claude Sonnet also has the best functional equivalence
score of 0. arXiv:2506. 00943v1 [cs.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Discussion on Performance Patterns
We observe a fundamental trade-off between precision and coverage in LLM-generated smart contracts, where models achieving higher precision scores tend to generate more focused implementations but sometimes miss implementing complete contract functionality <ref:2506.00943#pg2>. The impact of contract complexity is evident, as simple contracts with fewer legal positions see consistently high performance across all metrics and models <ref:2506. p. 6 The impact of contract complexity is particularly evident
in our results. Simple contracts with fewer legal positions,
such as the Shipper-carrier contract (3 obligations, no powers),
see consistently high performance across all metrics and
models. Simple contracts with fewer legal positions,
such as the Shipper-carrier contract (3 obligations, no powers),
see consistently high performance across all metrics and
models. However, common failure scenarios emerge with more
complex contracts:
• Incomplete implementation of contracts’ legal powers
and obligations, especially noticeable in GCDC ToS (7
powers), where only Claude 3.Where the paper says it
Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Development of a novel suite of metrics to quantify legal compliance based on modeling both legal and smart contracts as processes and comparing their behaviors allows for a
nuanced understanding of how different LLMs perform.
This enables automated and self-improving development workflows by providing a foundation for assessing code from any source, LLM or developer. -
Implementation of fitness, precision, and functional equivalence score (FES) metrics allows systems to compare the accuracy of smart contracts generated by different LLMs directly against natural language legal contracts. This enables users to determine
which LLM produces a smart contract most aligned with the legal ground truth.
-
The PN-based method to systematically calculate these metrics allows for a formal comparison between behaviors captured in a natural-language legal contract and those expressed in corresponding smart contracts. This provides a structured way to
quantify the degree of legal compliance for each LLM-generated smart contract, offering a nuanced understanding of how different LLMs perform.
-
The structured zero-shot prompting strategy based on the Plan-and-Solve (PS) approach, as detailed in Figure 5, allows AI systems to systematically extract key contract elements like
parties involved, powers and obligations
before generating code. This systematic approach addresses limitations encountered with previous prompting methods that facedtimeout and information capture issues.
-
The AI system can be improved to generate more robust smart contracts by incorporating best practices observed in successful LLM outputs, such as using OpenZeppelin library contracts like Pausable and Ownable, which are consistently used by high-performing models. This reduces the tendency of models like Llama 3.1 70b to produce
error-prone partial implementations that lack robust security features.
-
The system can be enhanced to better handle complex legal scenarios by exploring efficient methods for comparing Reachability Graphs (RGs), such as
using machine-learning techniques to identify common patterns and reduce the number of comparisons,
when dealing withcombinatorial growth in path comparisons
observed in complex contracts.
Abstract
Smart contracts can implement and automate parts of legal contracts, but ensuring their legal compliance remains challenging. Existing approaches such as formal specification, verification, and model-based development require expertise in both legal and software development domains, as well as extensive manual effort. Given the recent advances of Large Language Models (LLMs) in code generation, we investigate their ability to generate legally compliant smart contracts directly from natural language legal contracts, addressing these challenges. We propose a novel suite of metrics to quantify legal compliance based on modeling both legal and smart contracts as processes and comparing their behaviors. We select four LLMs, generate 20 smart contracts based on five legal contracts, and analyze their legal compliance. We find that while all LLMs generate syntactically correct code, there is significant variance in their legal compliance with larger models generally showing higher levels of compliance. We also evaluate the proposed metrics against properties of software metrics, showing they provide fine-grained distinctions, enable nuanced comparisons, and are applicable across domains for code from any source, LLM or developer. Our results suggest that LLMs can assist in generating starter code for legally compliant smart contracts with strict reviews, and the proposed metrics provide a foundation for automated and self-improving development workflows.
Sources
Related papers
- ArkEval: Benchmarking and Evaluating Automated CodeRepair for ArkTS
- TestPrism: Rethinking Test Evaluation Beyond a Single Reference
- Mining Workflow Graphs for Black-Box Boundary Testing of Conversational LLM Agents
- Falsification-Based Verification of LLM-Generated Optimization Models: Sound Test Batteries and Their Detection Limits
- GitSkills: A Dataset of Agent Skills on GitHub
- For Coding Agents, the Repository Matters