ChaosBench-Logic: A Benchmark for Logical and Symbolic Reasoning on Chaotic Dynamical Systems

arXiv:2601.01982 · cs.AI, cs.IT, math.IT · Submitted 2026-01-05 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "ChaosBench-Logic: A Benchmark for Logical and Symbolic Reasoning on Chaotic Dynamical Systems".

Jane: The gist: CHAOSBENCH-LOGIC introduces a benchmark to evaluate Large Language Model reasoning across 30 diverse chaotic dynamical systems using a unified first-order logic ontology,

Tom: First, who's behind it and why it matters.

Paper summary: Jane: So to wrap up the ChaosBench-Logic paper, it’s about creating a specific testbed for assessing LLM reasoning within chaotic dynamical systems using a first-order logic ontology. The main point is that while frontier models like GPT-four hit high per-item accuracy on individual facts, they consistently fail when asked to maintain global coherence or perform compositional reasoning across multiple steps or systems <ref:2601.01982#pg1>.

Lu: The authors emphasize that this failure stems from a lack of weak global constraint satisfaction, meaning the models can’t hold a consistent world model defined by those logical rules even when the rules are relatively simple.

Tom: The implication for me is clear: if you’re building an AI system meant to reason scientifically, you need more than just high accuracy on isolated questions; you need mechanisms that enforce consistency across long sequences of reasoning.

Meng: From a deployment standpoint, this means we can’t just rely on the latest model size or training data alone; we have to test them against these formal logical structures before they touch anything critical.

Jane: The paper suggests that this benchmark is a vital starting point for developing neuro-symbolic methods designed to instill that kind of structured, globally consistent belief system in AI.

Lu: It gives researchers a concrete way to diagnose where the symbolic reasoning breaks down and how to bridge the gap between fluid language and formal logic.

Tom: So, ChaosBench-Logic is basically showing us exactly what kind of logical failures we need to fix if we want LLMs to become reliable tools for complex scientific problem-solving.

Conclusion: Jane: So we’re wrapping up on ChaosBench-Logic, which is this new testbed for seeing how well large language models handle precise logic in chaotic systems.

Tom: Yeah it’s a benchmark that uses thirty different systems—like maps or differential equations—and tests the AI on them using first-order logic.

Lu: The core idea is taking deterministic chaos and trying to put it into a formal logical structure, which is where the authors focus their effort.

Meng: And they test questions across seven categories, so we're looking at things like multi-step math and asking if the AI can spot biases in the system.

Lalam: The results show that while these models get really good at answering single questions about a system, they struggle when you ask them to connect those facts across different systems or keep a long conversation coherent.

Jane: That means even though the individual answers are sometimes right, the AI can’t build that big picture of consistency we need for real scientific thinking.

Tom: I saw that ninety-one to ninety-four percent accuracy per item on things like chaotic or deterministic properties, which is solid for isolated facts. But they score zero on compositional items and the global coherence is fragile.

Lu: Exactly, it points to a weakness in what the models are actually doing internally—they aren't maintaining that consistent world model across all those different rules at once.

Meng: From a deployment angle, that fragility is a huge red flag because if you’re shipping something that needs to reason reliably, inconsistency at scale is just breaking things.

Jane: So what does this mean for us right now? It means we have a rigorous way to diagnose *why* the AI forgets the global rules when the problem gets complicated.

Tom: It gives us a tool to see if we can fix these failures by adding better symbolic structure or maybe some neuro-symbolic layers on top of the language.

Lu: The future work they propose is pretty ambitious, looking at multiple parameter regimes and combining that logic with actual numeric diagnostics from tools like equation solvers.

Meng: If they actually manage to bake that symbolic structure in, it could be a path toward models that don't just sound smart but actually reason reliably in complex domains.

Jane: We’ve seen how the current state of the art falls apart when we test it on these kinds of structured reasoning tasks. Next up, we’re looking at what this means for scientific AI generally and how to build better systems that stay grounded in reality.

Mohamed bin Zayed University of Artificial Intelligence (MBZUAI)

cs.AI, cs.IT, math.IT

Submitted: 2026-01-05

Updated: 2026-10-14

Comments: 7 pages, 0 figures, Accepted to AAAI-26 Bridge Program: Logical and Symbolic Reasoning in Language Models (camera-ready). v2: corrected bibliographic errors in the references

Journal ref: AAAI 2026 Bridge Program on Logical and Symbolic Reasoning in Language Models, Singapore, Jan 2026

Code: https://github.com/11NOel11/ChaosBench-Logic

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 79/100

The gist: The gist: CHAOSBENCH-LOGIC introduces a benchmark to evaluate Large Language Model reasoning across 30 diverse chaotic dynamical systems using a unified first-order logic ontology, revealing that

Key concepts

First-Order Logic (FOL) Ontology
This is the formal language used to define the properties of chaotic systems. It uses 11 specific logical predicates, like Chaotic(s) or Deterministic(s), to assign boolean truth values to different dynamical systems. This structure allows for precise, symbolic reasoning about how these systems relate to each other.
Global First-Order Logic Axiom System
This is a set of fixed rules that define the relationships between the logical predicates. For example, an axiom might state that if a system is Chaotic, it must also be Deterministic. This system tests whether the LLM can correctly apply these universal rules consistently across all parts of a complex problem.
Global Coherence
This refers to the model's ability to maintain a consistent worldview when answering multi-part questions or dialogues. It means the model doesn't contradict itself across different turns or segments of a reasoning task. The benchmark shows that current LLMs struggle significantly with this, often losing track of the overall context.
Compositional Reasoning
This is the ability to combine several small logical steps into one larger, correct conclusion. In this benchmark, it tests if an LLM can correctly apply multiple logical rules sequentially to solve a complex problem. The paper shows that frontier models fail at this, indicating a lack of robust long-term reasoning capabilities.

Terminology

Summary

The gist: CHAOSBENCH-LOGIC introduces a benchmark to evaluate Large Language Model reasoning across 30 diverse chaotic dynamical systems using a unified first-order logic ontology, revealing that frontier LLMs achieve high per-item accuracy but struggle with global coherence and compositional reasoning.

CHAOSBENCH-LOGIC Benchmark Structure

The benchmark evaluates LLM reasoning across 30 diverse dynamical systems spanning ODEs, maps, PDEs, neuronal and chemical models, and stochastic processes (Page 1). Each system is associated with semantic properties expressed as unary logical predicates from a fixed vocabulary P (Page 2). The benchmark consists of 621 questions over three diverse dynamical systems (Page 1).

Logical Ontology and Axiom System

CHAOSBENCH-LOGIC defines 11 unary predicates, each mapping a system s to a boolean (Page 3). These predicates include Chaotic(s), Deterministic(s), PosLyap(s) (positive largest Lyapunov exponent), Sensitive(s), StrangeAttr(s), PointUnpredictable(s), StatPredictable(s), QuasiPeriodic(s), Random(s) (true randomness / stochasticity), FixedPointAttr(s) and Periodic(s) (Page 3). A Global first-order logic Axiom System specifies a set Φ of global axioms encoding widely used implications in dynamical systems (Page 3). Examples include the axiom: ∀s: Chaotic(s) ⇒ Deterministic(s) (Page 3). Crucially, the system avoids converse rules, as it states: Design choice: no converse rules. Crucially, reverse implications are not assumed (Page 3).

Reasoning Task Families

The benchmark covers seven reasoning types (Page 4), including Atomic QA, Multi-Hop Implication Reasoning, Cross-System Analogical / Non-Analogical Reasoning, Counterfactual Parameter Reasoning, Bias Probes, Advanced Multi-Turn Dialogues and Hard Concept-Synthesis / Compositional (Page 4). The task families are designed to test specific failures:

Atomic QA Correctly recall system-level truth values (confident hallucination)

Multi-hop Apply directionality of implications (converse fallacy)

Counterfactual Reason about regime change vs invariance (overgeneralize chaos)

The dataset construction involves a four-stage pipeline including System encoding, Template-based generation with semantic constraints, and Human curation for correctness and ambiguity removal (Page 3).

Evaluation Metrics and Empirical Findings

Evaluation metrics are designed to separate local correctness from global coherence, introducing measures like Logical accuracy (Page 3) and Dialogue accuracy (Page 3). Initial experiments show that frontier LLMs such as GPT-4, Claude 3.5 Sonnet, Gemini 2.5 Flash, and the open-source LLaMA-3 70B achieve 91–94% per-item accuracy (Page 1). However, they still score 0% on compositional items and exhibit fragile global coherence: dialogue-level accuracy ranges from 53.1% (GPT-4 CoT) to 75.5% (LLaMA-3 zeroshot) (Page 1). The paper notes that Dialogue accuracy compounds errors: for a dialogue of length T, even an independent per-turn accuracy p yields an “all-correct” probability of approximately p T (Page 5).

Diagnostic Insights and Limitations

The central finding is weak global constraint satisfaction across all models, indicating they struggle to maintain a consistent global world model, even when the world model is as simple as a set of boolean predicates linked by Horn-style implications (Page 6). Chain-of-Thought (CoT) prompting is found to reduce both accuracy and dialogue-level coherence because it can introduce extra unsupported assumptions and amplify converse reasoning errors (Page 5). The benchmark's advantage lies in that inconsistency is measurable, allowing for diagnostics like tracking axiom violations within a dialogue (Page 5). Future work plans to extend the benchmark with "(i) multiple parameter regimes per system, (ii) items that require extracting regime information from equations or parameter values, and (iii) hybrid items that combine symbolic logic with toolverified numeric diagnostics" (Page 6).

The paper concludes that CHAOSBENCH-LOGIC supports progress toward scientific AI systems that maintain globally consistent beliefs, resist domain misconceptions, and integrate symbolic structure with fluent language understanding (Page 7). It is a rigorous testbed for diagnosing these failures and a foundation for developing neuro-symbolic approaches that improve scientific reasoning in LLMs (Page 1).

Improvements for AI systems

  1. Begin integrating symbolic verifiers into LLM architectures to enforce logical consistency; An LLM can propose answers while a symbolic module checks whether the answers violate Φ, triggering self-correction. This allows for Logic-guided decoding where outputs are constrained so that answers cannot jointly violate implications already committed within a dialogue.

  2. Develop training methodologies focused on consistency losses to improve belief state stability; this involves Fine-tuning can penalize axiom violations across turns, not only wrong final labels. This directly addresses the observed failure mode where models struggle to maintain a stable belief state across turns, as noted in Section 5.1.

  3. Implement logic-guided decoding specifically for multi-turn dialogues to enhance coherence; this ensures answers must not contradict earlier commitments under Φ, mitigating the belief drift that leads to lower DialogueAcc scores.

  4. Design a tool-augmented reasoning pipeline where LLMs can query symbolic modules for constraint checking, enabling them to handle Compositional questions deliberately combine multiple constraints, such as: (i) chaining two or more implications under Φ. This moves models beyond local-but-not-global reasoning.

  5. Incorporate bias probes directly into the fine-tuning process based on identified systematic biases; this allows for targeted mitigation of specific failures like the Chaos–Randomness Bias by training the model to correctly interpret predicate semantics.

Sources

Related papers