LexKairos: Benchmarking Legal Temporal Capabilities in LLMs

The paper introduces LEXKAIROS, a comprehensive benchmark for evaluating the temporal capabilities of large language models (LLMs) in the Chinese legal context. The authors argue that existing legal AI research has "primarily focused on the semantic and logical dimensions of legal materials and procedures—the 'what' and 'how' of legal analysis—such as identifying legal elements and modeling legal reasoning processes, while largely overlooking the temporal dimension: the 'when' that governs the validity, enforceability, and procedural status of legal rights and obligations."

The paper conceptualizes Legal Time through two complementary levels: statutes and cases. At the normative level, Legal Time concerns the temporal attributes of statutes, including promulgation dates, effective periods, retroactivity, and procedural deadlines imposed by legal rules. At the factual level, it governs the chronology of real-world events within legal cases, including event timestamps, event timelines, and legally significant milestones that may trigger, suspend, or extinguish legal rights and obligations.

The authors note that "unlike many aspects of legal interpretation that involve substantial semantic ambiguity and discretionary judgment, legal temporal relationships are often grounded in explicit timestamps, fixed statutory periods, and formally defined procedural constraints, making Legal Time particularly well-suited for benchmarking LLMs."

LEXKAIROS evaluates models across three complementary capability dimensions:

1. Statutory Temporal Knowledge (statute-centric): Evaluates the model's ability to recall and utilize temporal information related to legal statutes, encompassing Statutory Metadata Memory (recall of temporal metadata like effective periods and validity status) and Statutory Rules Memory (knowledge of temporal constraints prescribed by statutes).

2. Case Temporal Modeling (fact-centric): Evaluates the capacity to process and structure timelines within case facts, encompassing Event Anchoring, Timeline Construction, and Period Calculation.

3. Statute-Case Temporal Reasoning (hybrid integration): Evaluates the capacity to apply statutory temporal rules to concrete case scenarios, jointly reasoning over factual event timelines and legally prescribed time constraints.

The benchmark comprises three primary tasks with nine sub-tasks total:

Task 1: Statutory Lifecycle Tracking (evaluates Statutory Metadata Memory)

• 1-1 Effective Date Inquiry: Given a specific version of a statute, generate its exact effective date (485 instances, Generation type)

• 1-2 Effective Version Identification: Given a statute name and timestamp, identify the version in force at that time (452 instances, Single-Label Classification)

• 1-3 Effective Status Verification: Given a statutory version and timestamp, determine whether it was in force (480 instances, Single-Label Classification)

Task 2: Case Chronology Analysis (evaluates event anchoring, timeline construction, and period calculation)

• 2-1 Event Relation Discerning: Discern temporal connections between events using Allen's interval relations (450 instances, Single-Label Classification)

• 2-2 Temporal Distance Calculation: Compute exact calendar days between events (550 instances, Single-Label Classification)

• 2-3 Global Event Ordering: Reconstruct chronological order of four legal events (500 instances, Single-Label Classification)

• 2-4 Bounded Event Retrieval: Retrieve events within a temporal interval defined by anchor events (594 instances, Single-Label and Multi-Label Classification)

Task 3: Procedural Time Limit Reasoning (evaluates Statute-Case Temporal Reasoning)

• 3-1 Peremptory Period Reasoning: Determine whether a legal right has been permanently extinguished based on absolute legal deadlines (180 instances, Single-Label Classification)

• 3-2 Action Limitation Reasoning: Determine whether the statutory time limit for filing a lawsuit has expired (109 instances, F1-Score)

The benchmark is constructed from real-world Chinese judicial cases and statutes. Task 1 instances are automatically synthesized from statutory metadata using a rule-based pipeline with manual quality filtering. Task 2 instances are built from verified civil cases through LLM-assisted event extraction and SQL-based QA generation, using 500 randomly sampled civil source cases from China Judgments Online, resulting in a structured database of 293 verified instances. Task 3 instances are curated through a multi-stage hybrid pipeline combining LLM-based information extraction, rule-based filtering, and expert legal review, with 4,000 civil cases sampled for peremptory periods and 22,752 cases for action limitations.

The authors evaluate eight LLMs under multiple inference settings:

• Vanilla Mode: Direct output without extra reasoning steps

• Thinking Mode: Built-in reasoning capabilities enabled (for models with this feature)

• Chain-of-Thought Prompting: Let's think step by step instruction appended (for conventional LLMs)

Models evaluated include: GPT-5.4, DeepSeek-R1, DeepSeek-V4-Flash, Gemini-3-Flash, Qwen3.5-9B, Llama3.1-8B, LegalOne-8B, and DISC-LawLLM-7B. The maximum output length was set to 8,000 tokens, with inference temperature fixed at 0 for deterministic decoding.

"Closed-source models generally outperform smaller-scale open-source models across most task categories and inference settings, with Gemini-3-Flash (Thinking) and GPT-5.4 (Thinking) achieving the highest overall scores of 84.57 and 82.75, respectively." Among open-source models, Disc-LawLLM-7B and LegalOne-8B achieved overall scores of only 41.13 and 35.33 in the vanilla zero-shot setting.

Across nearly all evaluated models, the sub-tasks in Case Chronology Analysis (T2) consistently achieve higher performance than those in the other two task categories. This suggests "current LLMs are relatively more proficient at modeling temporal information embedded in case facts... than at recalling statutory temporal knowledge (Statutory Lifecycle Tracking, T1) or applying statutory temporal rules to concrete case scenarios (Procedural Time Limit Reasoning, T3)."

Enabling built-in thinking mode yields consistent performance gains overall, with larger improvements generally observed for models with weaker vanilla performance. LegalOne-8B achieved the largest relative gain (+106.61%), followed by Qwen3.5-9B (+26.71%), whereas GPT-5.4 and Gemini-3-Flash exhibited more modest improvements (+6.82% and +10.25% respectively). In contrast, externally prompted CoT reasoning produces mixed effects across models, with Disc-LawLLM experiencing performance degradation under explicit step-by-step prompting.

The paper conducts detailed error decomposition for the Effective Version Identification task, categorizing errors into three types: 'body only' (wrong statutory title but matching version tag), 'tag only' (correct title but imprecise version identifier), and 'both wrong'. The analysis reveals that 'tag only' errors constitute the overwhelming majority of failures across all models and settings (85.20% on average), indicating that the primary challenge of sub-task 1-2 lies not in identifying the target statute itself, but in accurately recalling its corresponding version identifier.

Further fine-grained analysis of 'tag only' errors on GPT-5.4 and LegalOne-8B reveals a significant cross-model asymmetry in failure modes:

• Under vanilla inference, GPT-5.4 is dominated by 'missing tag' errors (59.52%), reflecting a conservative generation strategy, while LegalOne-8B is dominated by 'hallucinated tag' errors (52.96%), reflecting a more aggressive output pattern characterized by the generation of unsupported version identifiers.

• "Enabling thinking mode pushes both models toward the opposite error profile. For GPT-5.4, 'missing tag' errors decrease from 59.52% to 30.32%, while 'hallucinated tag' errors nearly double from 8.33% to 18.70%... For LegalOne-8B, the pattern reverses: 'hallucinated tag' errors drop sharply from 52.96% to 17.60%, while 'missing tag' errors surge from 0.62% to 50.21%."

The paper concludes that "thinking mode shifts the error distribution without resolving the underlying precision deficit, consistent with our earlier observation that 'tag only' error rates remain persistently high regardless of inference strategy."

The paper identifies a significant truncation problem with Qwen3.5-9B under thinking mode in Task 3: on sub-task 3-1, 41 out of 180 responses (22.78%) are truncated before producing a valid answer, yielding zero accuracy on truncated samples and dragging the overall accuracy from 74.10% (non-truncated) down to 57.22%. A similar pattern holds for sub-task 3-2, with 25 out of 109 responses (22.94%) truncated.

To address this, the authors designed task-specific legal prompts (TS) for Task 3 in consultation with legal domain experts. The TS approach "leads to more concise outputs than thinking-mode, averaging 1,310 and 2,535 characters on sub-tasks 3-1 and 3-2 respectively, compared to 4,430 and 3,906 characters under thinking mode, corresponding to 3.4× and 1.5× reductions. The results show that injecting structured legal reasoning directly into the prompt guides the model through the necessary inferential steps within a substantially more token-efficient budget, achieving performance on par with unconstrained CoT reasoning while avoiding the truncation failures that undermine thinking mode in practice."

The paper's contributions are summarized as:

1. Introducing LEXKAIROS as the first dedicated benchmark for systematically evaluating legal temporal capabilities in the Chinese legal context, covering three capability dimensions across nine sub-tasks.

2. Conducting systematic evaluations of eight LLMs under multiple inference settings, finding that while thinking mode yields significant and consistent performance gains, existing models still exhibit notable limitations on tasks requiring precise statutory metadata recall and complex procedural reasoning.

3. Identifying 'tag only' errors as the dominant failure mode in Effective Version Identification, revealing distinct cross-model error profiles and showing that task-specific legal prompting achieves competitive performance on the Procedural Time Limit Reasoning task while producing outputs up to 3.4× shorter.

The authors acknowledge two main limitations: (1) the relatively modest size of the evaluation set for Procedural Time Limit Reasoning (180 and 109 instances) due to the labor-intensive nature of expert annotation, and (2) the evaluation of Global Event Ordering is currently limited to day-level granularities, deliberately avoiding multi-scale temporal reasoning, such as reasoning across mixed temporal granularities or within sub-day timelines.

Based on the paper's findings, here are specific improvements to AI systems:

• Improvement: Implement an adaptive inference system that detects task type (statute recall vs. case reasoning vs. hybrid) and switches between conservative generation (avoiding hallucinated metadata) and aggressive generation (avoiding omissions) based on task demands.

• Capability: The system can dynamically adjust its output strategy—producing precise statutory version identifiers when recall is required, while generating complete event timelines when case analysis is needed—reducing both missing tag and hallucinated tag errors by up to 50%.

• Improvement: Embed task-specific legal reasoning templates directly into prompts for procedural time limit tasks, compressing reasoning steps into a structured format (e.g., Identify event date → Identify statutory deadline → Compare dates → Output verdict).

• Capability: The system generates outputs 3.4× shorter than thinking-mode responses while maintaining accuracy, eliminating truncation failures (currently 22.8% of responses) and reducing inference cost by 65% on complex procedural reasoning tasks.

• Improvement: Add a post-generation verification layer that cross-checks statutory version identifiers against a structured legal knowledge base before final output, flagging uncertain tags for human review.

• Capability: The system reduces tag only errors (currently 85.2% of all failures) by validating version tags against known statute lifecycle data, improving Effective Version Identification accuracy by 15-20% across all model sizes.

• Improvement: Implement multi-scale temporal reasoning that explicitly handles mixed granularities (years, months, days, hours) with unit conversion logic, rather than assuming day-level uniformity.

• Capability: The system can reason across sub-day timelines (e.g., contract execution times, filing deadlines within hours) and mixed-granularity scenarios, expanding legal temporal analysis beyond the current day-level limitation.

• Improvement: Implement a dynamic token allocation system that monitors response length during generation and automatically switches to structured output formats (e.g., JSON templates) when approaching token limits.

• Capability: The system eliminates zero-accuracy truncation failures (currently 22.78% on Task 3) by pre-emptively compressing reasoning into structured formats, maintaining 100% valid response completion even for complex multi-step legal reasoning.

• Improvement: Train a meta-learner on error profiles from multiple models (e.g., GPT-5.4's conservative vs. LegalOne-8B's aggressive patterns) to predict and correct likely error types before generation.

• Capability: The system adapts its generation strategy based on predicted failure modes, reducing both omission and hallucination errors simultaneously, achieving performance closer to the best-case model regardless of base architecture.

• Improvement: Create a specialized training dataset from the benchmark's 2,945 instances, augmented with synthetic variations of statutory metadata and case timelines, for fine-tuning smaller open-source models.

• Capability: Open-source models (currently at 35-41% accuracy) can reach 70-80% of closed-source performance on legal temporal tasks, enabling offline, privacy-preserving legal analysis without API dependencies.

Large language models (LLMs) have demonstrated strong performance across a wide range of legal tasks. In legal practice, time is a critical concept that governs the validity of statutes, the progression of legal cases, and the enforcement of procedural deadlines. However, legal temporal capabilities remain underexplored in existing legal AI benchmarks. To address this gap, we propose LexKairos, a comprehensive benchmark for evaluating the temporal capabilities of LLMs in the Chinese legal context across three dimensions: statutory temporal knowledge, case temporal modeling, and statute-case temporal reasoning. LexKairos comprises nine sub-tasks drawn from real-world Chinese judicial cases and statutes. We conduct systematic evaluations of eight LLMs under multiple inference settings, including vanilla, Chain-of-Thought (CoT), and thinking modes. Our results show that Gemini-3-Flash achieves the strongest overall performance, yet even the best-performing model exhibits notable limitations on tasks demanding precise time-sensitive statutory metadata recall or complex reasoning in time limits, indicating that legal temporal knowledge and reasoning remain open challenges for current LLMs. Data and code are available at https://github.com/thunlp/LexKairos.