Legal Compliance Evaluation of Smart Contracts Generated By Large Language Models
summary
The gist
Smart contracts can implement and automate parts of legal contracts, but ensuring their legal compliance remains challenging <ref:2506.00943#pg2> Given the recent advances of Large Language Models…
In short
The study investigates if Large Language Models (LLMs) can accurately generate legally compliant smart contracts from natural language legal documents. Researchers used novel metrics—fitness, precision, and Functional Equivalence Score—to compare four different LLMs across five legal contracts. Results show that while all models produce valid code, larger models generally yield higher compliance scores. This suggests LLMs are useful for generating starter code requiring strict human review.
Key concepts
- Fitness
- This metric measures how many specific behaviors or clauses from the original legal contract are actually present and strictly matched in the generated smart contract. It quantifies the degree of coverage of the required legal obligations within the resulting code.
- Precision
- Precision assesses how much extra, irrelevant behavior an LLM adds to its output beyond what is explicitly required by the legal document. High precision means less unnecessary code is generated, focusing only on relevant contract elements.
- Functional Equivalence Score (FES)
- The FES evaluates overall functional similarity by checking if both the legal contract and the smart contract result in the same legally equivalent state at their conclusion. It focuses on the ultimate outcome rather than just matching every single step in sequence.
Terminology used across episodes
This episode discusses
- Legal Compliance Evaluation of Smart Contracts Generated By Large Language Models · Paper Radio
- The Llama 3 Herd of Models · Paper Radio
The paper
Legal Compliance Evaluation of Smart Contracts Generated By Large Language Models · Read on arXiv
Chanuka Wijayakoon, Hai Dong, H.M.N. Dilum Bandara, Zahir Tari, Anurag Soin
School of Computing Technologies, RMIT University · CSIRO's Data61 Institute for Cyber-Physical Systems Research Centre
Smart contracts can implement and automate parts of legal contracts, but ensuring their legal compliance remains challenging. Existing approaches such as formal specification, verification, and model-based development require expertise in both legal and software development domains, as well as extensive manual effort. Given the recent advances of Large Language Models (LLMs) in code generation, we investigate their ability to generate legally compliant smart contracts directly from natural language legal contracts, addressing these challenges. We propose a novel suite of metrics to quantify legal compliance based on modeling both legal and smart contracts as processes and comparing their behaviors. We select four LLMs, generate 20 smart contracts based on five legal contracts, and analyze their legal compliance. We find that while all LLMs generate syntactically correct code, there is significant variance in their legal compliance with larger models generally showing higher levels of compliance. We also evaluate the proposed metrics against properties of software metrics, showing they provide fine-grained distinctions, enable nuanced comparisons, and are applicable across domains for code from any source, LLM or developer. Our results suggest that LLMs can assist in generating starter code for legally compliant smart contracts with strict reviews, and the proposed metrics provide a foundation for automated and self-improving development workflows.
DOI: 10.1109/ICBC64466.2025.11114661
Transcript
Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.
Tom: I'm Tom.
Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.
Tom: We're looking at the paper: "Legal Compliance Evaluation of Smart Contracts Generated By Large Language Models".
Jane: Here's the gist of the paper. Smart contracts can implement and automate parts of legal contracts, but ensuring their legal compliance remains challenging <ref:2506.00943#pg2> Given the recent advances of Large Language Models (LLMs) in code generation, we investigate their ability to generate legally compliant smart contracts directly from natural language legal contracts, addressing these challenges <ref:2506.00943#pg2>.
Tom: We propose a novel suite of metrics to quantify legal compliance based on modeling both legal and smart contracts as processes and comparing their behaviors <ref:2506.00943#pg2>.
Jane: We focus on the alignment between the behaviors captured in a natural-language legal contract and those expressed in the corresponding smart contract, using this as a proxy to assess LLMs’ capability to generate legally compliant code <ref:2506.00943#pg2>.
Tom: To facilitate this comparison, we adopt a contracts-as-processes view
12: , modeling both legal and smart contracts as Petri Nets (PNs)
13: to analyze their behaviors <ref:2506.00943#pg2>.
Paper summary: Tom: So wrapping up this discussion on "Legal Compliance Evaluation of Smart Contracts Generated By Large Language Models," the paper shows that while large language models can generate code, we still need a rigorous way to measure how well that code actually adheres to the legal requirements <ref:2506.00943#pg2>.
Jane: The main implication is that instead of just hoping the AI gets it right, we now have metrics like fitness and precision that let us quantify the gap between what’s in a legal document and what’s actually in a smart contract <ref:2506.00943#pg1>.
Lu: This is significant because it moves us away from just saying "the AI did well" to having objective data on *how* well it performed <ref:2506.00943#pg2>.
Meng: For deployment, this means we can start building systems where the compliance score from these metrics is a necessary gate before any code goes live <ref:2506.00943#pg1>.
Tom: Exactly. And the work by Chanuka Wijayakoon and her team gives us a foundation for automated and self-improving development workflows, provided we use these tools correctly <ref:2506.00943#pg2>.
Jane: It’s about providing a way to help the AI learn and improve its legal generation skills by giving it concrete feedback based on how it aligns with our defined legal processes <ref:2506.00943#pg1>.
Lu: Ultimately, this paper is about building the infrastructure for legally sound AI-generated code, which is a necessary step before we can really rely on these systems for anything substantial <ref:2506.00943#pg2>.
Conclusion: Tom: So we're wrapping up on this paper, "Legal Compliance Evaluation of Smart Contracts Generated By Large Language Models." Essentially, they've set up a way to actually check if those smart contracts the large language models write are actually following the law.
Jane: That makes sense. Before this, you could just hope the AI did a good job with legal stuff, but now they’re giving us specific ways to measure that gap between what’s in the contract and what’s in code.
Lu: The core idea here is modeling both a legal document and a smart contract as processes—using something called Petri Nets—so they can compare their behaviors directly, not just look at the text superficially.
Meng: I'm interested because it moves beyond just syntax checking. It tries to capture the actual *logic* of the legal requirements, which is where most AI fails when things get messy and complex.
Lalam: I can confirm that they introduced three main metrics: fitness, which checks how many behaviors from the legal contract are in the smart contract; precision, which looks at how much extra behavior you added that wasn't actually required; and functional equivalence score, which just checks if both documents end up in the same final state.
Tom: Those metrics sound like they’re giving us a real score instead of a vague "it probably worked" feeling. I’m curious about the results themselves.
Jane: Yeah, the experimental setup involved taking five legal contracts and asking four different large language models to generate twenty smart contracts from them. And what they found was that while every model wrote syntactically correct code, there was a big difference in how compliant they were.
Lu: They found that larger models generally showed higher compliance scores, which suggests the size or training data of the language model plays a role in understanding legal nuance better.
Meng: That's interesting—so bigger models are better at this specific task, even though I’d bet on needing way more testing on real-world traffic to know if that holds up when things break at scale.
Lalam: They did find specific examples, like for a contract called GCDC ToS, where one model hit a high score of zero point five for both fitness and functional equivalence. That’s concrete data from the comparison.
Tom: So, it sounds like the authors are showing that while LLMs are great at writing code structure, we need these kinds of process-based metrics to move toward actually trusting them with anything legally sensitive.
Jane: Exactly. It's not about replacing lawyers or developers; it’s about giving us a tool to review and refine the AI’s output much more effectively before deployment.
Lu: The implication is that we can start building automated workflows where the AI generates a first draft, gets scored by these metrics, and then only needs human experts to focus on the areas with low scores.
Meng: If they can automate that scoring process, it could drastically reduce the manual review time for generating smart contracts. That’s something I think would actually make production feasible.
Tom: So we're moving from just being amazed by what the AI writes to having a measurable, quantifiable standard for legal correctness in code generation.
Jane: Right. And that foundation is what opens up the door for future work, looking at how these metrics can be used in self-improving systems where the models learn from their compliance failures over time.
More episodes
- 2505.16254-Human-AI Writing as Distributed Cognition: Reassessing Collaborative Writing Theories in the Age of LLMs
- 2506.10913-Choreographic Quick Changes: First-Class Location (Set) Polymorphism
- 2507.14812-A Black-Box Approach for Exogenous Replenishment in Online Resource Allocation
- 2507.15448-Frames over finite fields with respect to the Galois inner product and their connections to self-orthogonal and LCD codes
- 2508.21572-NewsReX: An Open-Source Multi-Framework for Neural News Recommendation
- 2509.17028-Impact of Packetization on Network Calculus Analysis
- 2510.17342-AoA Services in 5G Networks: A Framework for Real-World Implementation and Systematic Testing
- 2511.04088-Efficient and rate-optimal list-decoding in the presence of minimal feedback
- 2511.17264-Notes on Stack Machines and Quantum Stack Machines
- 2601.15318-On the closest balanced game