Can ChatGPT Perform Reasoning Using the IRAC Method in Analyzing Legal Scenarios Like a Lawyer?
cs.CL
Submitted: 2023-10-23
Updated: 2026-10-28
Comments: EMNLP 2023 Findings
Journal ref: 2023.findings-emnlp.929
DOI: 10.18653/v1/2023.findings-emnlp.929
Code: https://github.com/christinakang/SIRAC
License: http://creativecommons.org/licenses/by/4.0/
The gist: Large Language Models (LLMs), such as ChatGPT, have drawn a lot of attentions recently in the legal domain due to its emergent ability to tackle a variety of legal tasks.
Terminology
Abstract
Large Language Models (LLMs), such as ChatGPT, have drawn a lot of attentions recently in the legal domain due to its emergent ability to tackle a variety of legal tasks. However, it is still unknown if LLMs are able to analyze a legal case and perform reasoning in the same manner as lawyers. Therefore, we constructed a novel corpus consisting of scenarios pertain to Contract Acts Malaysia and Australian Social Act for Dependent Child. ChatGPT is applied to perform analysis on the corpus using the IRAC method, which is a framework widely used by legal professionals for organizing legal analysis. Each scenario in the corpus is annotated with a complete IRAC analysis in a semi-structured format so that both machines and legal professionals are able to interpret and understand the annotations. In addition, we conducted the first empirical assessment of ChatGPT for IRAC analysis in order to understand how well it aligns with the analysis of legal professionals. Our experimental results shed lights on possible future research directions to improve alignments between LLMs and legal experts in terms of legal reasoning.
Sources
- Can GPT-3 Perform Statutory Reasoning?
- A Survey on In-context Learning
- Shortcut Learning of Large Language Models in Natural Language Understanding
- LegalBench: Prototyping a Collaborative Benchmark for Legal Reasoning
- CUAD: An Expert-Annotated NLP Dataset for Legal Contract Review
- Factoring Statutory Reasoning as Language Understanding Challenges
- Towards Reasoning in Large Language Models: A Survey
- Natural Language Processing in the Legal Domain
- Personalisation within bounds: A risk taxonomy and policy framework for the alignment of large language models with personalised feedback
- Augmented Language Models: a Survey
- REFINER: Reasoning Feedback on Intermediate Representations
- Unlocking Practical Applications in Legal Domain: Evaluation of GPT for Zero-Shot Semantic Annotation of Legal Texts
- Large Language Models Can be Lazy Learners: Analyze Shortcuts in In-Context Learning
- MAUD: An Expert-Annotated Legal NLP Dataset for Merger Agreement Understanding
- Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- Legal Prompting: Teaching a Language Model to Think Like a Lawyer
Related papers
- ALARM: Audio-Language Alignment for Reasoning Models
- ThinkSum: Probabilistic reasoning over sets using large language models
- Coherence boosting: When your pretrained language model is not paying enough attention
- From Facts to Insights: A Persona-Driven Dual Memory Framework and Dataset for Role-Playing Agents
- Improving General Role-Playing Agents via Psychology-Grounded Reasoning and Role-Aware Policy Optimization
- The Art of Asking: Multilingual Prompt Optimization for Synthetic Data