When Do LLMs Apply the Wrong Law? Diagnosing LLM Failures in Temporal Legal Reasoning

arXiv:2608.14610 · cs.AI · Submitted 2026-07-08 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: Today's paper: "When Do LLMs Apply the Wrong Law? Diagnosing LLM Failures in Temporal Legal Reasoning".

Jane: The gist The LLMs exhibit a strong bias toward applying the most recently enacted law, regardless of when the legally relevant facts occurred,

Tom: First, who's behind it and why it matters.

Paper summary: Jane: So we’re wrapping up our look at "When Do LLMs Apply the Wrong Law? Diagnosing LLM Failures in Temporal Legal Reasoning." The authors are looking at a huge set of fifty-one thousand nine hundred fifty-six Chinese civil judgments and testing nine different large language models to see why they mess up when it comes to time periods in the law > Page one >

Tom: That’s right. The paper explains that this isn't just about not knowing old laws; it’s about how the reinforcement learning training makes the AI policy collapse, meaning it just defaults to what seems most current instead of checking the case timing > Page two > <ref:2608.14610#pg1>

Lu: The core idea is that TALD—that’s Temporal Applicable-Law Determination—requires models to figure out which version of a statute applies based on when the relevant events actually happened, not just when you ask the question > Page two > <ref:2608.14610#pg1>

Meng: From a deployment standpoint, this convergence on newer statutes is a big problem because real life involves cases from years ago. If we ship that today, it won't work for historical legal questions > Page three > <ref:2608.14610#pg1>

Meng: If we can’t solve this policy misalignment, then whatever accuracy gains we see in their general problem-solving abilities won't matter for complex legal tasks > Page three > <ref:2608.14610#pg1>

Tom: So to wrap up, the authors show that the issue isn't a lack of knowledge about prior versions; it’s a policy collapse caused by reinforcement learning training that makes the AI default to newer statutes regardless of case facts.

Conclusion: Tom: So, we're wrapping up our look at "When Do LLMs Apply the Wrong Law? Diagnosing LLM Failures in Temporal Legal Reasoning." The paper investigates why large language models have a strong bias toward newer statutes, even when the facts of a case happened much earlier.

Jane: That’s right. The authors looked at fifty-one thousand nine hundred fifty-six Chinese civil judgments to test nine different models and found that the problem isn't just ignorance of old laws; it's how reinforcement learning training makes the AI policy collapse into applying the most recent law, regardless of when things actually happened.

Lu: The task they define is Temporal Applicable-Law Determination, or TALD. It requires these models to figure out which version of a statute applies based on case facts and the date of those events.

Meng: From a deployment standpoint, that convergence on newer statutes is a big problem because real life involves cases from years ago. If we ship that today, it won't work for historical legal questions.

Lalam: I’m seeing the data show that even when we give the models specific hints about retroactivity or timing—like a strong temporal-law hint—the uncertainty actually goes up during reasoning, which suggests those hints might force them to reconsider their default bias toward newer law.

Tom: That’s interesting. So, the paper isn't just saying "make the model smarter" but rather we have to change how we guide its existing capabilities, not just feed it more general legal knowledge.

Jane: Exactly. The implication for us is that when we build these systems, it’s less about boosting their general reasoning ability and more about aligning their specific policies with the temporal rules of the law itself.

Lu: It points toward a need to engineer task-specific reasoning policies so they don't get stuck in that deterministic loop where they only apply the current statutes.

Meng: If we can’t solve this policy misalignment, then whatever accuracy gains we see in their general problem-solving abilities won't matter for complex legal tasks.

Lalam: I’m confident that by focusing on those version-related reasoning paths instead of just boosting raw reasoning, we can actually get better results on these temporal determination tasks.

Tom: Next time, we’re going to look at how the authors suggest tweaking those hints—the weak versus strong temporal-law hints—and see what that does to the model's behavior.

Institute of Science Tokyo University of Osaka Institute of Science and Technology Center of Juris-Informatics ROIS-DS

cs.AI

Submitted: 2026-07-08

Updated: 2026-10-15

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 79/100

The gist: The gist The LLMs exhibit a strong bias toward applying the most recently enacted law, regardless of when the legally relevant facts occurred, which suggests that reinforcement-learning-shaped

Key concepts

Temporal Applicable-Law Determination (TALD): This is the task where an AI must figure out which specific version of a statute applies to a legal case. It involves looking at the facts of the case and matching them against different versions of a law that have changed over time, requiring temporal reasoning.
Policy-Entropy Collapse: This occurs when reinforcement learning training makes an AI's decision-making process too predictable and narrow. Instead of exploring many different ways to solve a problem, the model converges on one specific path—in this case, defaulting to applying the most recent law—reducing its ability to consider alternatives.
Temporal-Law Hints: These are explicit instructions given to an LLM that guide it toward correct temporal reasoning. When models receive strong hints about timing and retroactivity, they show increased exploration of different legal versions, suggesting guidance helps them overcome their default bias.

Terminology

Accuracy — 25× in this paper · explained in 1 episode
Benchmark — 10× in this paper · explained in 2 episodes
Reinforcement Learning — 9× in this paper · explained in 4 episodes
Temporal Applicable-Law Determination — 7× in this paper
Entropy Collapse — 4× in this paper · explained in 2 episodes
Memorization — 4× in this paper · explained in 2 episodes
Generalization — 3× in this paper · explained in 13 episodes
Reinforcement Learning (RL) — 3× in this paper · explained in 25 episodes
Large Language Models (LLMs) — 2× in this paper · explained in 8 episodes
Papers — 2× in this paper · explained in 3 episodes
Robustness — 2× in this paper · explained in 5 episodes
Systematic Bias — 2× in this paper · explained in 3 episodes
Thinking mode — 2× in this paper · explained in 2 episodes
Benchmarking — 1× in this paper · explained in 4 episodes
Calibration — 1× in this paper · explained in 7 episodes
Core — 1× in this paper · explained in 2 episodes
Distribution Shift — 1× in this paper · explained in 9 episodes
Downstream Tasks — 1× in this paper · explained in 2 episodes
Generalizability — 1× in this paper · explained in 6 episodes
Instruction Tuning — 1× in this paper · explained in 3 episodes
Mixture of Experts — 1× in this paper · explained in 2 episodes
Reasoning Traces — 1× in this paper · explained in 3 episodes
Trace — 1× in this paper · explained in 2 episodes

Summary

The gist The LLMs exhibit a strong bias toward applying the most recently enacted law, regardless of when the legally relevant facts occurred, which suggests that reinforcement-learning-shaped explicit reasoning reduces diversity in reasoning paths, causing models to converge on applying the current law. <ref:2608.14610#pg4>

The Temporal Applicable-Law Determination Task

The paper introduces the Temporally Applicable-Law Determination (TALD) task, which requires models to identify which statutory version governs a given case based on case facts and the date of relevant events <ref:2608.14610#pg3>. The objective of TALD is to maximize TALD accuracy, defined by the matching accuracy between the ground truth and prediction <ref:2608.14610#pg3>. The task involves given a collection of statutes N = N = n, where each statute i ∈ N may have multiple versions due to temporal legal evolution <ref:2608.14610#pg4>. Given a query (F, Q), the TALD task aims to identify a sequence of legal citations y = [y1,..., yn] where yi denotes the correct version of statue i that is legally applicable to answering Q based on F <ref:2608.14610#pg4>.

Key Findings on LLM Failures

The empirical investigation reveals four key findings regarding why LLMs fail at temporal legal reasoning <ref:2608.14610#pg3>. First, LLMs exhibit a strong and systematic bias toward applying the most recently enacted law, regardless of when the legally relevant facts occurred <ref:2608.14610#pg4>. Second, this bias does not stem from ignorance of prior statutory versions, nor from a failure to understand that laws have temporal scope <ref:2608.14610#pg4>. Third, "we identify reinforcement learning (RL) as a key mechanism: while it improves general reasoning ability, it simultaneously reduces the diversity of reasoning paths, causing models to converge almost exclusively on applying the current law <ref:2608.14610#pg4>. Fourth, this leads to a counterintuitive result where models with stronger general reasoning ability tend to perform worse on temporal legal reasoning" <ref:2608.14610#pg4>.

Analysis of Knowledge and Reasoning Deficiencies

The analysis probes whether failures stem from knowledge or reasoning capability <ref:2608.14610#pg6>. Probe A, testing statutory-text memorization, shows that most models can reproduce both predicted and gold provisions with high similarity to the official text <ref:2608.14610#pg6>. However, Probe B on temporal-effect rule knowledge indicates that models generally perform well on this rule-knowledge probe <ref:2608.14610#pg6>. This suggests that TALD failure is not primarily caused by the absence of explicit knowledge about temporal-effect doctrines <ref:2608.14610#pg6>.

The Role of General Reasoning and Policy Alignment

Hypothesis 2, examining general reasoning capability, found that stronger general reasoning does not solve TALD; engaging it more strongly often makes performance worse <ref:2608.14610#pg6>. This pattern contradicts the hypothesis that TALD failure is merely due to insufficient general reasoning ability <ref:2608.14610#pg6>. Furthermore, Hypothesis 3 suggests that TALD failure is closely tied to task-specific reasoning-policy misalignment <ref:2608.14610#pg6>. Domain expertise hints were shown to redirect reasoning-oriented models toward the correct temporal-applicability policy, whereas stronger general reasoning alone does not guarantee improvement <ref:2608.14610#pg6>.

Policy-Entropy Collapse Mechanism

The paper hypothesizes that TALD failure is related to policy-entropy collapse, where RL training reduces exploratory behavior and makes the policy increasingly deterministic <ref:2608.14610#pg7>. This mechanism suggests that newer statutes are often more salient in contemporary legal texts and semantically close to their predecessors <ref:2608.14610#pg8>. Behavioral evidence shows that "Interventions that improve TALD are generally accompanied by higher entropy in version-related CoT spans, suggesting that successful TALD requires task-directed diversity over temporal-law alternatives rather than merely more explicit reasoning" <ref:2608.14610#pg8>.

Conclusion and Implications

The findings highlight the need for future legal LLMs to align reasoning not only with general problem-solving ability, but also with domain-specific applicability policies <ref:2608.14610#pg10>. The work provides concrete guidance for improving LLM performance in temporally grounded legal reasoning <ref:2608.14610#pg4>. Limitations include the dataset's scope and the mechanistic interpretation being behavioral rather than causal <ref:2608.14610#pg10>. The study complements downstream legal prediction tasks by diagnosing whether models can first determine which version of law should govern the case <ref:2608.14610#pg8>.

A System Prompts with Hint Levels

The paper details three prompt settings for temporal-law hints: Default Prompt (No Hint), Weak Temporal-Law Hint, and Strong Temporal-Law Hint. The strong hint explicitly instructs the model to Pay close attention to the timing of events in the case and to the law’s temporal applicability / intertemporal effect, in particular issues of retroactivity. This demonstrates that explicit guidance is necessary for reasoning-oriented models when they need direction toward the correct temporal-law policy <ref:2608.14610#pg6>. The paper concludes that successful TALD correction is accompanied by reopening versionrelated reasoning paths, but larger entropy is not itself the objective <ref:2608.14610#pg8>. The results show that Hint perturbations increase version-related exploration when TALD improves <ref:2608.14610#pg8>. The final table shows that Hint perturbations increase version-related exploration when TALD improves <ref:2608.14610#pg8>.

Improvements for AI systems

Make LLMs capable of temporal applicable-law determination (TALD) by building a benchmark that systematically tests whether models can identify, given a case’s facts and the date of the relevant events, which statutory version governs the case. This allows for controlled evaluation of temporal applicability identification.
Develop models that exhibit accurate legal reasoning by addressing the misaligned task-specific reasoning policy by providing domain expertise hints. Specifically, Domain expertise hints can redirect reasoning-oriented models toward the correct temporal-applicability policy, improving TALD accuracy even when general reasoning is insufficient.
Implement a mechanism to prevent reinforcement learning from reducing diversity in legal reasoning paths by ensuring that interventions aim for higher entropy in version-related CoT spans. This encourages reopening exploration over temporal-law alternatives rather than converging on a narrow newest-law trajectory.

Abstract

Legal reasoning requires identifying which version of a statute applies to a case, typically based on when the legally relevant events occurred rather than when the case is adjudicated. Yet whether large language models (LLMs) can reliably perform such temporal legal reasoning remains largely unexplored. We formalize this task as Temporal Applicable-Law Determination (TALD), and construct a TALD-Bench, a benchmark of 51,956 Chinese civil judgments spanning the enactment of the Civil Code. Evaluating nine advanced LLMs, we identify three key findings. First, models exhibit a strong bias toward newer law, achieving over 70% accuracy on new-law cases but below 25% on old-law cases. Second, models demonstrate substantial knowledge of statutory versions and temporal-effect rules, yet struggle to apply this knowledge to cases. Stronger general reasoning performance does not consistently translate into better TALD accuracy, whereas temporal-law hints yield substantial improvements, suggesting a bottleneck in how models deploy their existing capabilities. Third, temporal-law hints also increase uncertainty during statutory-version reasoning, suggesting that they may help models reconsider their default preference for newer law. Our findings highlight the importance of adapting how LLMs apply existing capabilities, rather than relying solely on improvements in legal knowledge or general reasoning.