ChaosBench-Logic v2: Evaluating LLM Logical Reasoning over Dynamical Systems at Scale

Name: ChaosBench-Logic v2: Evaluating LLM Logical Reasoning over Dynamical Systems at Scale
Uploaded: 2026-10-17T13:22:54.141262+00:00
Duration: 852 s
Description: Researchers created ChaosBench-Logic v2, a large benchmark testing LLM reasoning over 165 dynamical systems using first-order logic. A new evaluation protocol, CARE, reveals that LLMs struggle with parameter-dependent dynamics and prior collapse. Findings show proprietary models excel at consistency tasks, while open-source models dominate indicator diagnostics.

Researchers created ChaosBench-Logic v2, a large benchmark testing LLM reasoning over 165 dynamical systems using first-order logic. A new evaluation protocol, CARE, reveals that LLMs struggle with parameter-dependent dynamics and prior collapse. Findings show proprietary models excel at consistency tasks, while open-source models dominate indicator diagnostics.

Open the full episode