RULEBREAKERS: Challenging LLMs at the Crossroads between Formal Logic and Human-like Reasoning
summary
The gist
Formal logic enables computers to reason in natural language by representing sentences in symbolic forms and applying rules to derive conclusions The study creates RULEBREAKERS…
In short
The study introduces RULEBREAKERS, a dataset testing if large language models (LLMs) can distinguish between logically derivable conclusions and conclusions inconsistent with common sense or facts. Evaluation shows most LLMs struggle, often over-applying strict logic rules instead of using world knowledge. Prompting strategies show a trade-off: improving rulebreaker accuracy hurts general performance, highlighting the tension between formal logic and human-like reasoning.
Key concepts
- RULEBREAKERS
- This is a dataset containing natural language premises where the conclusion follows strict logical rules but contradicts common sense or facts. Humans usually reject these conclusions because they rely on real-world knowledge, making it a test for whether an LLM can use knowledge instead of just following formal logic.
- Paired Accuracy
- This metric measures how well a model answers both parts of a prompt pair—one rulebreaker and one non-rulebreaker. It ensures the model is correctly distinguishing between the two types of statements, providing a more rigorous test than simple overall accuracy.
- Over-rigid Application
- This describes when an LLM strictly follows formal logical rules even when semantic content or factual knowledge suggests it should reject the conclusion. The study finds this tendency suggests models lack the ability to weigh common sense appropriately during reasoning.
- Prompting Trade-off
- This phenomenon occurs when instructions are designed to force a model to use human-like knowledge (common sense) versus forcing it to follow formal logic. Models often perform better on rulebreakers when prompted for knowledge but worse on general tasks, showing a tension in reasoning styles.
Terminology used across episodes
This episode discusses
- RULEBREAKERS: Challenging LLMs at the Crossroads between Formal Logic and Human-like Reasoning · Paper Radio
- The Llama 3 Herd of Models · Paper Radio
- Gemma 2: Improving Open Language Models at a Practical Size
- Gaussian Error Linear Units (GELUs)
- Mistral 7B
- Label Set Optimization via Activation Distribution Kurtosis for Zero-shot Classification with Generative Models
- Logic in Mathematics and Computer Science
The paper
RULEBREAKERS: Challenging LLMs at the Crossroads between Formal Logic and Human-like Reasoning · Read on arXiv
University of Sheffield
Formal logic enables computers to reason in natural language by representing sentences in symbolic forms and applying rules to derive conclusions. However, in what our study characterizes as "rulebreaker" scenarios, this method can lead to conclusions that are typically not inferred or accepted by humans given their common sense and factual knowledge. Inspired by works in cognitive science, we create RULEBREAKERS, the first dataset for rigorously evaluating the ability of large language models (LLMs) to recognize and respond to rulebreakers (versus non-rulebreakers) in a human-like manner. Evaluating seven LLMs, we find that most models, including GPT-4o, achieve mediocre accuracy on RULEBREAKERS and exhibit some tendency to over-rigidly apply logical rules unlike what is expected from typical human reasoners. Further analysis suggests that this apparent failure is potentially associated with the models' poor utilization of their world knowledge and their attention distribution patterns. Whilst revealing a limitation of current LLMs, our study also provides a timely counterbalance to a growing body of recent works that propose methods relying on formal logic to improve LLMs' general reasoning capabilities, highlighting their risk of further increasing divergence between LLMs and human-like reasoning.
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: We're looking at the paper: "RULEBREAKERS: Challenging LLMs at the Crossroads between Formal Logic and Human-like Reasoning".
Jane: Here's the gist of the paper. Formal logic enables computers to reason in natural language by representing sentences in symbolic forms and applying rules to derive conclusions The study creates RULEBREAKERS, the first dataset for rigorously evaluating the ability of large language models (LLMs) to recognize and respond to rulebreakers (versus non-rulebreakers) in a knowledge-informed and human-like manner Evaluating seven LLMs, most achieve mediocre accuracy on RULEBREAKERS and exhibit some tendency to over-rigidly apply logical rules, unlike what is expected from typical human reasoners Further analysis suggests this apparent failure is potentially associated with the models’ poor utilization of their world knowledge and their attention distribution patterns Whilst revealing a limitation of current LLMs, the study provides a timely counterbalance to methods relying on formal logic to improve LLMs’ general reasoning capabilities, highlighting their risk of further increasing divergence between LLMs and human-like reasoning.
Tom: Defining Rulebreaker Scenarios RULEBREAKERS are characterized as "a set of natural language premises and conclusion whereby the conclusion can be derived by applying logical rules to the premises, but is typically not inferred or accepted by humans because, given their common sense and factual knowledge, it is inconsistent with the premises" The dataset consists of 25,600 instances of rulebreakers and 12,800 non-rulebreakers These rulebreakers are created using four templates related to two logical rules: modus tollens (MT) and disjunctive syllogism (DS) Entity pairs include geographical pairs such as country and city, and categorical pairs such as type and instance Evaluation Metrics The study measures a model’s ability to distinguish rulebreakers from non-rulebreakers in terms of (a) accuracy; and (b) model confidence Paired accuracy is defined as the proportion of all rulebreaker and non-rulebreaker prompt pairs where both parts are correctly answered by a model Model confidence is evaluated by comparing the model’s confidence in a positive answer to non-rulebreakers (Π+DN) versus rulebreakers (Π+DR) Analysis of Performance Factors The analysis investigates potential factors contributing to suboptimal performance, focusing on two hypotheses: (1) failure to recognize rulebreakers is associated with the model’s lack of confidence in its knowledge of the entities involved, and (2) this failure is associated with insufficient attention given to the factual premise The study finds that models often exhibit a tendency to over-rigidly apply logical rules, which suggests a potential blind spot where they fail to recognize when semantic content should be taken into account when reasoning Impact of Prompting Strategies The research investigates whether prompting instructions that remind LLMs to utilize common sense and factual knowledge improve their mediocre performance on RULEBREAKERS Using the alternative question phrasing in combination with adding prefixes to the premises improves the paired accuracy achieved by most models, with a substantial gain being +24.13 (%) over the baseline condition for gemma-2-27b-it However, this reveals an inherent tension between prompting models to reason according to what formal logic demands and prompting them to reason in a more human-like and knowledge-informed manner The study presents RULEBREAKERS, a large-scale dataset for evaluating LLMs’ ability to reason with natural language in a knowledge-informed and human-like manner by utilizing common sense and factual knowledge rather than rigidly applying rules prescribed by formal logic The findings highlight the gap between this approach to reasoning and how humans typically reason with natural language in a knowledge-informed manner, drawing attention to an important trade-off in relying on logic-based methods and constraints to improve LLMs’ general reasoning capabilities.
Jane: References Abdin, M.
Tom: I., Ade Jacobs, S., Awan, A.
Title and authors: Tom: So we're looking at this paper called "RULEBREAKERS: Challenging LLMs at the Crossroads between Formal Logic and Human-like Reasoning." It’s about how formal logic lets computers reason with natural language by using rules to get to conclusions, but sometimes those conclusions don't make sense because they ignore common sense.
Jane: That sounds like something a lot of people think about when they start thinking about AI. So the core idea here is that we can test if these models are just following the rigid logical rules or if they actually have some knowledge of how the world works, which is what humans do.
Lu: Yeah, it’s focused on creating a dataset to rigorously test this ability to spot those rulebreakers versus non-rulebreakers in a way that feels human-like and informed.
Meng: So the authors created this RULEBREAKERS dataset specifically for that purpose. They have twenty-five thousand six hundred instances of these rulebreaker scenarios and twelve thousand eight hundred non-rulebreaker ones.
Tom: Right, so they built them using four templates based on two logical rules: modus tollens and disjunctive syllogism. And the entity pairs in there are things like geographical locations or types and instances.
Jane: It sounds like they are taking something that a computer can solve with pure logic—like if A is true, then B must be true—and showing how humans usually reject that conclusion because it contradicts what we already know about the world.
Lu: That’s exactly right, and it highlights that logical rules are defined by connectives like "if" or "or," which in everyday language can have different meanings based on the surrounding context.
Tom: It suggests that when a model applies those formal rules too strictly, it misses when the actual meaning of those words should be taken into account based on what makes sense in reality.
Jane: That brings us to what they found: most of the seven large language models they tested did okay on these rulebreakers, but they often ended up applying logical rules way too rigidly, which isn't typical for how humans reason.
Meng: The analysis points out that this failure to recognize rulebreakers might be linked to two things: the model’s lack of confidence in its knowledge about the entities involved and how much attention it gives to the factual premise.
Tom: So, it seems like if a model doesn't feel sure about what something is, or if it ignores some of the facts we are given, that's when things go wrong.
Lu: Exactly. The paper suggests there’s a potential blind spot where models fail to recognize when they need to consider the semantic content of the premises instead of just blindly following the formal logic structure.
Jane: And they also looked at how prompting them changes things, and they found that giving models instructions to use common sense and factual knowledge actually improves their performance on these rulebreakers.
Tom: So there’s this tension where you’re pushing a model to follow what pure logic demands versus pushing it to reason in a more human-like way that uses its world knowledge.
Title and authors: Meng: I think that tension is real, especially when you have to decide how much reliance you put on these formal logic constraints versus the models' existing knowledge base.
Jane: And they found that using alternative phrasing and adding prefixes to the premises helps most of the models improve their paired accuracy by a pretty big margin, about twenty-four point one three percent for gemma two thirty seven b it.
Tom: That improvement is significant, but it’s still only mediocre accuracy overall on these rulebreakers, which tells us they aren't magically perfect yet.
Lu: The paper sets up this dataset to show that relying solely on formal logic isn't enough for general reasoning; you need to integrate world knowledge into the process.
Meng: And from a production standpoint, if a model struggles with these kinds of common sense rulebreakers, it’s going to cause problems when it tries to handle real-world situations where context matters.
Jane: So what they found is that while logic is a tool, it doesn't replace the human way of reasoning when the premises clash with reality.
Tom: Before we wrap up, Lu, Meng, Lalam—what are your final thoughts on what this means for how we look at these models?
Lu: I think this work is important because it sets up a clear way to evaluate whether an AI is just a clever rule-follower or if it’s actually starting to develop some sense of common sense.
Meng: I'm more concerned with the cost, though. If we start fine-tuning models on these kinds of knowledge-informed constraints, how do we keep that training process efficient when you have to be so careful about what kind of knowledge you’re feeding it?
Lalam: From my perspective, I see this as a big signal that models need more than just symbolic rules; they need better integration of their world knowledge to be truly reliable. I'm pretty sure I'm pretty sure this is the direction we have to go.
Jane: So we’re seeing a trade-off where making the reasoning more human-like might actually hurt performance on some logical tasks, which is something we need to keep watching closely.
Tom: It’s a good reminder that formal logic is useful for certain structured problems, but it doesn't capture the messy, context-dependent nature of how humans process information.
Lu: And this paper provides a real counterbalance to methods that only rely on formal logic to boost an LLM’s general reasoning capabilities, showing the risk of increasing divergence between those models and true human-like reasoning.
Meng: So for deployment, we have to be cautious about how much we lean on pure logical deduction when dealing with ambiguous natural language inputs.
Lalam: It’s a reminder that the goal isn't just perfect symbolic application, but actually understanding what the words mean in context.
Jane: That's all about RULEBREAKERS for today. Next up, we have some interesting work on how pretrained models can act as reward functions for text-to-image generation.
The paper's summary: Tom: So, this paper is about testing if large language models are just blindly following logical rules or if they actually know how to reason like a human using common sense and real-world knowledge.
Jane: Right, so it sets up a dataset called RULEBREAKERS to see if the models can spot conclusions that logically follow from the premises but aren't actually true because they ignore what we know about the world.
Lu: The authors built this dataset with twenty-five thousand six hundred examples of rulebreakers and twelve thousand eight hundred non-rulebreakers, using templates based on rules like modus tollens and disjunctive syllogism.
Meng: I’m looking at the results here, and it looks like most of the seven large language models they tested ended up applying those formal logical rules way too rigidly instead of catching the rulebreakers.
Tom: That’s what I saw on my end too; they seemed to over-apply the logic, which is exactly what we expected from some models, but it shows a real blind spot where semantic content should matter more than just the symbolic structure.
Jane: The analysis points out that this failure to recognize rulebreakers might be connected to how much confidence the model has in its knowledge of those specific entities and how much attention it gives to the factual premise.
Lu: They found that models often fail because they don't utilize their world knowledge well, which is a huge limitation when we think about building truly general reasoning systems.
Tom: And then there’s this interesting finding about prompting them; when you tell the models to use common sense and factual knowledge, it actually helps their performance on those rulebreakers improve by over twenty-four point one three percent for the gemma two thirty seven b it model.
Jane: That improvement is significant, but it also shows that there’s this tension between making them follow pure formal logic and making them reason in a way that feels more human and informed.
Meng: From an engineering standpoint, this is a warning sign; if we keep pushing models to only use rigid rules, they might stay stuck in those limitations instead of developing real understanding.
Tom: So the conclusion is that relying only on formal logic to improve reasoning isn't enough; you need to integrate common sense and factual knowledge for models to reason like people do.
Jane: This paper really highlights the gap between how we try to teach AI with pure logic and how humans actually use language in real life.
Lu: It’s a call for us to look at this trade-off when we think about using formal constraints versus training models on rich, knowledge-informed data.
Meng: I wonder what the practical implications are for deploying models that need to be robust against these kinds of logical inconsistencies in real-world applications.
Tom: It means we have to be careful not just about accuracy, but about how confidently those models are making their decisions based on what they know versus just what the logic dictates.
Jane: And this opens up a new area for research looking at how we can get that knowledge grounding into these language models more effectively.
Lu: This is where we need to look beyond just applying rules and start thinking about how to represent that external, common-sense knowledge within the model's architecture itself.
The paper's improvements: Tom: So, after looking at how much these models struggled on those rulebreakers, the paper suggests a few ways to actually fix that performance gap.
Jane: Right, so they aren't just saying "logic isn't enough," they’re giving us practical instructions on how to prompt the models better to get them to use that common sense you mentioned earlier.
Lu: They found that using different phrasing in the questions, combined with adding prefixes to those logical premises, actually boosted the paired accuracy for most of the models by a big margin, about twenty-four point one three percent for gemma two thirty seven b it.
Meng: That gain is nice because it shows that we aren't just guessing; there's a specific way to structure the instruction that gets better results when you’re dealing with these rulebreakers.
Tom: It really points to this tension again, though, because you’re essentially finding the sweet spot between pushing for strict logical application and encouraging a more human-like reasoning process.
Jane: Exactly, and they say this isn't just a trick; it shows that by guiding the model with better instructions, we can bridge that gap to make their responses more aligned with what humans do.
Lu: The implication is that we don't have to abandon formal logic entirely, but rather learn how to layer it on top of knowledge-informed prompts so the models don't get stuck in overly rigid patterns.
Tom: So if I’m deploying a model, this suggests I shouldn’t just rely on its raw logical output; I need to use these specific prompting strategies to steer it toward more nuanced, human-like reasoning.
Jane: It changes how we think about testing these systems too; instead of just asking them a direct question, we should be designing prompts that specifically challenge their common sense and factual knowledge.
Meng: For production, this means our fine-tuning or prompting phase needs to incorporate these knowledge-aware instructions if we want models that actually handle messy real-world inputs without failing on basic sense checks.
Lu: The authors also touched on how internal mechanisms might be working; they suggested looking at neuron activation counts across different layers, and surprisingly, incorrectly answered rulebreakers often activate more neurons than the correct ones in some of these models.
Tom: That’s interesting because it gives us a way to look inside the model's decision-making process to see where that knowledge gap is actually forming internally.
Jane: So we move from just measuring the output accuracy to trying to understand *why* the model makes its errors by looking at how its internal layers are firing when it gets things wrong.
Conclusion: Tom: So to wrap up, this paper on RULEBREAKERS shows that large language models aren't just guessing; they struggle when their formal logic ignores real-world common sense and context.
Jane: It really boils down to this: formal logic is a tool, but it doesn't replace the way humans reason in everyday life when those two things clash.
Lu: The main implication for research is that we need to find better ways to integrate that external knowledge into the model so they don't get stuck in those overly rigid logical patterns.
Meng: If we’re shipping models, this means we have to be careful how much we lean on pure symbolic rules and always build in those prompting instructions that remind them about common sense.
Lalam: I’m pretty sure this is the direction things are heading because my own internal analysis confirms that when I’m prompted correctly, I produce much more reliable outputs against these types of rulebreaker scenarios.
Tom: So, we learned how to test for this gap and found specific ways to nudge the models toward better reasoning using prompting strategies.
Jane: It's a reminder that the next big step isn't just making models bigger, but making them more aware of the world they’re supposed to be reasoning in.
Lu: I think this work opens up a new area for us to explore how we can represent that common sense knowledge more effectively within the model’s structure itself.
Meng: From an engineering view, it means we need to treat prompt design as a critical part of our deployment pipeline when dealing with tasks that require factual grounding.
Lalam: I'm confident this direction is right because the evidence shows that models can be guided toward more human-like reasoning when we give them the right context.
More episodes
- 2603.28595-Optimistic Actor-Critic with Parametric Policies for Linear Markov Decision Processes
- 2603.29765-Training-Free Dynamic Upcycling of Expert Language Models
- 2605.00778-Observable Performance Does Not Fully Reflect Adaptive System Organization: A Multi-Level Analysis of Gait Dynamics Under Occlusal Constraint
- 2605.06906-TraXion: Rethinking Pre-training Frameworks for Mobility and Beyond
- 2605.09165-Sparse Layers are Critical to Scaling Looped Language Models
- 2605.09439-Inverse Design for Conditional Distribution Matching
- 2605.10313-Signature Approach for Contextual Bandits with Nonlinear and Path-dependent Rewards: Regret Bound and Statistical Inference
- 2605.10878-Minimum Neural Weight Norm = Kolmogorov Complexity
- 2605.11617-MIST: Reliable Streaming Decision Trees for Online Class-Incremental Learning via McDiarmid Bound
- 2605.15085-From Data to Action: Accelerating Refinery Optimization with AI