Legal Compliance Evaluation of Smart Contracts Generated By Large Language Models

arXiv:2506.00943 · cs.SE, cs.AI · Submitted 2025-06-01 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "Legal Compliance Evaluation of Smart Contracts Generated By Large Language Models".

Jane: Here's the gist of the paper. Smart contracts can implement and automate parts of legal contracts, but ensuring their legal compliance remains challenging <ref:2506.00943#pg2> Given the recent advances of Large Language Models (LLMs) in code generation, we investigate their ability to generate legally compliant smart contracts directly from natural language legal contracts, addressing these challenges <ref:2506.00943#pg2>.

Tom: We propose a novel suite of metrics to quantify legal compliance based on modeling both legal and smart contracts as processes and comparing their behaviors <ref:2506.00943#pg2>.

Jane: We focus on the alignment between the behaviors captured in a natural-language legal contract and those expressed in the corresponding smart contract, using this as a proxy to assess LLMs’ capability to generate legally compliant code <ref:2506.00943#pg2>.

Tom: To facilitate this comparison, we adopt a contracts-as-processes view

12: , modeling both legal and smart contracts as Petri Nets (PNs)

13: to analyze their behaviors <ref:2506.00943#pg2>.

Paper summary: Tom: So wrapping up this discussion on "Legal Compliance Evaluation of Smart Contracts Generated By Large Language Models," the paper shows that while large language models can generate code, we still need a rigorous way to measure how well that code actually adheres to the legal requirements <ref:2506.00943#pg2>.

Jane: The main implication is that instead of just hoping the AI gets it right, we now have metrics like fitness and precision that let us quantify the gap between what’s in a legal document and what’s actually in a smart contract <ref:2506.00943#pg1>.

Lu: This is significant because it moves us away from just saying "the AI did well" to having objective data on *how* well it performed <ref:2506.00943#pg2>.

Meng: For deployment, this means we can start building systems where the compliance score from these metrics is a necessary gate before any code goes live <ref:2506.00943#pg1>.

Tom: Exactly. And the work by Chanuka Wijayakoon and her team gives us a foundation for automated and self-improving development workflows, provided we use these tools correctly <ref:2506.00943#pg2>.

Jane: It’s about providing a way to help the AI learn and improve its legal generation skills by giving it concrete feedback based on how it aligns with our defined legal processes <ref:2506.00943#pg1>.

Lu: Ultimately, this paper is about building the infrastructure for legally sound AI-generated code, which is a necessary step before we can really rely on these systems for anything substantial <ref:2506.00943#pg2>.

Conclusion: Tom: So we're wrapping up on this paper, "Legal Compliance Evaluation of Smart Contracts Generated By Large Language Models." Essentially, they've set up a way to actually check if those smart contracts the large language models write are actually following the law.

Jane: That makes sense. Before this, you could just hope the AI did a good job with legal stuff, but now they’re giving us specific ways to measure that gap between what’s in the contract and what’s in code.

Lu: The core idea here is modeling both a legal document and a smart contract as processes—using something called Petri Nets—so they can compare their behaviors directly, not just look at the text superficially.

Meng: I'm interested because it moves beyond just syntax checking. It tries to capture the actual *logic* of the legal requirements, which is where most AI fails when things get messy and complex.

Lalam: I can confirm that they introduced three main metrics: fitness, which checks how many behaviors from the legal contract are in the smart contract; precision, which looks at how much extra behavior you added that wasn't actually required; and functional equivalence score, which just checks if both documents end up in the same final state.

Tom: Those metrics sound like they’re giving us a real score instead of a vague "it probably worked" feeling. I’m curious about the results themselves.

Jane: Yeah, the experimental setup involved taking five legal contracts and asking four different large language models to generate twenty smart contracts from them. And what they found was that while every model wrote syntactically correct code, there was a big difference in how compliant they were.

Lu: They found that larger models generally showed higher compliance scores, which suggests the size or training data of the language model plays a role in understanding legal nuance better.

Meng: That's interesting—so bigger models are better at this specific task, even though I’d bet on needing way more testing on real-world traffic to know if that holds up when things break at scale.

Lalam: They did find specific examples, like for a contract called GCDC ToS, where one model hit a high score of zero point five for both fitness and functional equivalence. That’s concrete data from the comparison.

Tom: So, it sounds like the authors are showing that while LLMs are great at writing code structure, we need these kinds of process-based metrics to move toward actually trusting them with anything legally sensitive.

Jane: Exactly. It's not about replacing lawyers or developers; it’s about giving us a tool to review and refine the AI’s output much more effectively before deployment.

Lu: The implication is that we can start building automated workflows where the AI generates a first draft, gets scored by these metrics, and then only needs human experts to focus on the areas with low scores.

Meng: If they can automate that scoring process, it could drastically reduce the manual review time for generating smart contracts. That’s something I think would actually make production feasible.

Tom: So we're moving from just being amazed by what the AI writes to having a measurable, quantifiable standard for legal correctness in code generation.

Jane: Right. And that foundation is what opens up the door for future work, looking at how these metrics can be used in self-improving systems where the models learn from their compliance failures over time.

Chanuka Wijayakoon, Hai Dong, H.M.N. Dilum Bandara, Zahir Tari, Anurag Soin

School of Computing Technologies, RMIT University · CSIRO's Data61 Institute for Cyber-Physical Systems Research Centre

cs.SE, cs.AI

Submitted: 2025-06-01

Updated: 2025-06-01

Comments: Accepted for publication at IEEE International Conference on Blockchain and Cryptocurrency (ICBC) 2025

Journal ref: 2025 IEEE International Conference on Blockchain and Cryptocurrency (ICBC), Pisa, Italy, 2025, pp. 1-9

DOI: 10.1109/ICBC64466.2025.11114661

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 85/100

The gist: Smart contracts can implement and automate parts of legal contracts, but ensuring their legal compliance remains challenging <ref:2506.00943#pg2> Given the recent advances of Large Language Models…

Key concepts

Fitness
This metric measures how many specific behaviors or clauses from the original legal contract are actually present and strictly matched in the generated smart contract. It quantifies the degree of coverage of the required legal obligations within the resulting code.
Precision
Precision assesses how much extra, irrelevant behavior an LLM adds to its output beyond what is explicitly required by the legal document. High precision means less unnecessary code is generated, focusing only on relevant contract elements.
Functional Equivalence Score (FES)
The FES evaluates overall functional similarity by checking if both the legal contract and the smart contract result in the same legally equivalent state at their conclusion. It focuses on the ultimate outcome rather than just matching every single step in sequence.

Terminology

Summary

Smart contracts can implement and automate parts of legal contracts, but ensuring their legal compliance remains challenging <ref:2506. p. 0

Where the paper says it

soin@student. Abstract—Smart contracts can implement and automate parts of legal contracts, but ensuring their legal compliance remains challenging.

00943#pg2> Given the recent advances of Large Language Models (LLMs) in code generation, we investigate their ability to generate legally compliant smart contracts directly from natural language legal contracts, addressing these challenges <ref:2506. p. 0
Where the paper says it

Existing approaches such as formal specification, verification, and model-based development require expertise in both legal and software development domains, as well as extensive manual effort. Given the recent advances of Large Language Models (LLMs) in code generation, we investigate their ability to generate legally compliant smart contracts directly from natural language legal contracts, addressing these challenges.

00943#pg2>. We propose a novel suite of metrics to quantify legal compliance based on modeling both legal and smart contracts as processes and comparing their behaviors <ref:2506. p. 0
Where the paper says it

Given the recent advances of Large Language Models (LLMs) in code generation, we investigate their ability to generate legally compliant smart contracts directly from natural language legal contracts, addressing these challenges. We propose a novel suite of metrics to quantify legal compliance based on modeling both legal and smart contracts as processes and comparing their behaviors.

00943#pg2>.

Methodology Overview

We focus on the alignment between the behaviors captured in a natural-language legal contract and those expressed in the corresponding smart contract, using this as a proxy to assess LLMs’ capability to generate legally compliant code <ref:2506.00943#pg2>. To facilitate this comparison, we adopt a contracts-as-processes view [12], modeling both legal and smart contracts as Petri Nets (PNs) [13] to analyze their behaviors <ref:2506. p. 0

Where the paper says it

We focus on the alignment between the behaviors captured in a natural-language legal contract and those expressed in the corresponding smart contract, and use it as a proxy to assess LLMs’ capability to generate legally compliant code. To facilitate this comparison, we adopt a contracts-as-processes view [12], modeling both legal and smart contracts as Petri Nets (PNs) [13] to analyze their behaviors.

00943#pg2>. We introduce a novel suite of metrics for legal compliance: 1) fitness, measuring how many behaviors from the legal contract are captured in the smart contract; 2) precision, assessing how many behaviors in the smart contract are relevant to the legal contract; and 3) functional equivalence score (FES), providing a flexible assessment of the alignment between <ref:2506. p. 0
Where the paper says it

To facilitate this comparison, we adopt a contracts-as-processes view [12], modeling both legal and smart contracts as Petri Nets (PNs) [13] to analyze their behaviors. We introduce a novel suite of metrics for legal compliance: 1) fitness, measuring how many behaviors from the legal contract are captured in the smart contract; 2) precision, assessing how many behaviors in the smart contract are relevant to the legal contract; and 3) functional equivalence score (FES), providing a flexible assessment of the alignment between the

00943#pg2>.

Metrics Definition

The fitness metric measures how closely the behaviors of a smart contract (SC) match those of the legal contract (LC) based on strict event equivalence <ref:2506.00943#pg2>. Fitness is defined as the ratio of behaviors that strictly match between the contracts to the total number of behaviors in the legal contract, formalized as F(SC, LC) = R ∩ Q / R <ref:2506. p. 4

Where the paper says it

Additionally, when multiple behaviors in a smart contract map to a behavior in the legal contract, they are counted as one. The fitness F(SC, LC) of a smart contract to the legal contract is defined as the ratio of behaviors that strictly match between the contracts to the total number of behaviors in the legal contract.

00943#pg2>. Precision measures how much extra behavior a smart contract introduces beyond those in the legal contract, defined as Pr(SC, LC) = (R ∩ Q)∗ / Q <ref:2506. p. 4
Where the paper says it

5, respectively, A better captures the set of expected behaviors from the legal contract. Precision Precision measures how much extra behavior a smart contract introduces beyond those in the legal contract.

00943#pg2>. The Functional Equivalence Score (FES) measures overall functional similarity by comparing whether at the end of corresponding behaviors, both the legal and smart contracts reach legally equivalent states, regardless of event sequences <ref:2506. p. 4
Where the paper says it

Functional Equivalence Score (FES) FES measures the overall functional similarity between the legal contract and the smart contract, relaxing the strict event equivalence requirement of the fitness metric. It compares whether at the end of corresponding behaviors, both the legal and smart contracts reach legally equivalent states, even if the event sequences differ.

00943#pg2>.

Experimental Setup and Findings

We selected four LLMs—GPT-4o, Gemini 1.5 Pro, Claude 3. p. 0

Where the paper says it

While LLMs can produce syntactically correct code [ 7], generating smart contracts from natural-language legal agreements GPT-4o Gemini 1. 5 Pro Claude 3.

5 Sonnet, and Llama 3. p. 0
Where the paper says it

5 Pro Claude 3. 5 Sonnet Llama 3.

1 70b Instruct—to generate 20 smart contracts based on five legal contracts <ref:2506. p. 0
Where the paper says it

We propose a novel suite of metrics to quantify legal compliance based on modeling both legal and smart contracts as processes and comparing their behaviors. We select four LLMs, generate 20 smart contracts based on five legal contracts, and analyze their legal compliance.

00943#pg2>. We observed that while all LLMs generate syntactically correct code, there is significant variance in their legal compliance with larger models generally showing higher levels of compliance <ref:2506. p. 0
Where the paper says it

We select four LLMs, generate 20 smart contracts based on five legal contracts, and analyze their legal compliance. We find that while all LLMs generate syntactically correct code, there is significant variance in their legal compliance with larger models generally showing higher levels of compliance.

00943#pg2>. For the GCDC contract, Claude 3. p. 4
Where the paper says it

It is defined as the ratio of behaviors in the smart contract that are functionally equivalent to those in the legal contract, regardless of strict event matching: F ES(SC, LC) = |(R ∩ Q) ∗ | |R| (3) TABLE II OVERVIEW OF LEGAL CONTRACTS Contract Name Obligations Powers Behaviors* GCDC Terms of Service 0 7 12 Meat sale 3 3 12 Pizza delivery 3 2 3 Shipper-Carrier 3 0 3 Transactive energy 3 3 24 * Behaviors in the reachability graph of the legal contract’s PN. TABLE III LARGE ANGUAGE MODELS AND THEIR CHARACTERISTICS Model Knowledge Context Max Open Cutoff Window Output Source Claude 3.

5 Sonnet achieved the highest fitness score at 0. p. 5
Where the paper says it

3) Compliance Metrics: Table IV shows the compliance metrics for selected contracts. For GCDC, we observe: • Claude Sonnet has the highest fitness score at 0.

5 and the best functional equivalence score of 0. p. 5
Where the paper says it

Without this functionality, LLM outputs do not reach the same final state as the legal contract. • Claude Sonnet also has the best functional equivalence score of 0.

5 <ref:2506. p. 0
Where the paper says it

arXiv:2506. 00943v1 [cs.

00943#pg2>.

Discussion on Performance Patterns

We observe a fundamental trade-off between precision and coverage in LLM-generated smart contracts, where models achieving higher precision scores tend to generate more focused implementations but sometimes miss implementing complete contract functionality <ref:2506.00943#pg2>. The impact of contract complexity is evident, as simple contracts with fewer legal positions see consistently high performance across all metrics and models <ref:2506. p. 6

Where the paper says it

The impact of contract complexity is particularly evident in our results. Simple contracts with fewer legal positions, such as the Shipper-carrier contract (3 obligations, no powers), see consistently high performance across all metrics and models.

00943#pg2>. Conversely, more complex contracts demonstrate common failure scenarios like incomplete implementation of legal powers and obligations in GCDC ToS <ref:2506. p. 6
Where the paper says it

Simple contracts with fewer legal positions, such as the Shipper-carrier contract (3 obligations, no powers), see consistently high performance across all metrics and models. However, common failure scenarios emerge with more complex contracts: • Incomplete implementation of contracts’ legal powers and obligations, especially noticeable in GCDC ToS (7 powers), where only Claude 3.

00943#pg2>.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Development of a novel suite of metrics to quantify legal compliance based on modeling both legal and smart contracts as processes and comparing their behaviors allows for a nuanced understanding of how different LLMs perform. This enables automated and self-improving development workflows by providing a foundation for assessing code from any source, LLM or developer.

  2. Implementation of fitness, precision, and functional equivalence score (FES) metrics allows systems to compare the accuracy of smart contracts generated by different LLMs directly against natural language legal contracts. This enables users to determine which LLM produces a smart contract most aligned with the legal ground truth.

  3. The PN-based method to systematically calculate these metrics allows for a formal comparison between behaviors captured in a natural-language legal contract and those expressed in corresponding smart contracts. This provides a structured way to quantify the degree of legal compliance for each LLM-generated smart contract, offering a nuanced understanding of how different LLMs perform.

  4. The structured zero-shot prompting strategy based on the Plan-and-Solve (PS) approach, as detailed in Figure 5, allows AI systems to systematically extract key contract elements like parties involved, powers and obligations before generating code. This systematic approach addresses limitations encountered with previous prompting methods that faced timeout and information capture issues.

  5. The AI system can be improved to generate more robust smart contracts by incorporating best practices observed in successful LLM outputs, such as using OpenZeppelin library contracts like Pausable and Ownable, which are consistently used by high-performing models. This reduces the tendency of models like Llama 3.1 70b to produce error-prone partial implementations that lack robust security features.

  6. The system can be enhanced to better handle complex legal scenarios by exploring efficient methods for comparing Reachability Graphs (RGs), such as using machine-learning techniques to identify common patterns and reduce the number of comparisons, when dealing with combinatorial growth in path comparisons observed in complex contracts.

Abstract

Smart contracts can implement and automate parts of legal contracts, but ensuring their legal compliance remains challenging. Existing approaches such as formal specification, verification, and model-based development require expertise in both legal and software development domains, as well as extensive manual effort. Given the recent advances of Large Language Models (LLMs) in code generation, we investigate their ability to generate legally compliant smart contracts directly from natural language legal contracts, addressing these challenges. We propose a novel suite of metrics to quantify legal compliance based on modeling both legal and smart contracts as processes and comparing their behaviors. We select four LLMs, generate 20 smart contracts based on five legal contracts, and analyze their legal compliance. We find that while all LLMs generate syntactically correct code, there is significant variance in their legal compliance with larger models generally showing higher levels of compliance. We also evaluate the proposed metrics against properties of software metrics, showing they provide fine-grained distinctions, enable nuanced comparisons, and are applicable across domains for code from any source, LLM or developer. Our results suggest that LLMs can assist in generating starter code for legally compliant smart contracts with strict reviews, and the proposed metrics provide a foundation for automated and self-improving development workflows.

Sources

Related papers