Explanation Generation for Contradiction Reconciliation with LLMs

arXiv:2603.22735 · cs.CL · Submitted 2026-03-24 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "Explanation Generation for Contradiction Reconciliation with LLMs".

Jane: The gist The task of reconciling contradictory statements through natural language explanations is an underexplored but key aspect of intelligence in tasks ranging from conversations to scientific discovery.

Tom: First, who's behind it and why it matters.

Paper summary: Tom: So we’ve talked about the core idea: this paper, "Explanation Generation for Contradiction Reconciliation with LLMs," looks at how large language models handle contradictions. The central thesis is that existing NLP work only focuses on detecting conflicts or choosing which statement to discard.

Jane: Instead, this research introduces a new task called reconciliatory explanation generation where the model has to create natural language explanations that actually make contradictory statements compatible. They do this by defining success through two specific qualities: effectiveness and coherence.

Lu: Effectiveness means the premise combined with the explanation must entail the hypothesis, and coherence means that the explanation itself shouldn't contradict your initial premise. It sets a formal way to measure if a generated explanation is actually helpful or just plausible sounding.

Meng: They build this task by taking existing datasets like MultiNLI and filtering them down to instances where contradiction was the most common label, creating a subset called ChaosNLI-MNLI-C with two hundred seventy-five instances.

Tom: They then prompt these models to generate an explanation that resolves that apparent contradiction, specifically requiring both effectiveness and coherence in the output. The evaluation relies on judge models checking if the premise plus explanation leads to the hypothesis, and if the explanation itself is coherent with the premise.

Jane: This setup allows them to use standard NLI three-way judgments—checking relationships between premise plus explanation and hypothesis, between premise and explanation, and between the explanation and premise—to automatically score how well an AI handled that contradiction.

Lu: They are repurposing existing NLI datasets rather than building a completely new foundation from scratch for this reconciliation task. They focus on linguistic nuances like implicatures and ambiguities of the statements being reconciled, which is a key difference from previous work.

Meng: From my perspective, this is interesting because it’s not just about simple fact checking; it’s about modeling that messy, nuanced human-like way we bridge conflicting ideas in complex scenarios.

Tom: Exactly. It moves the focus from simple arbitration to generating explanatory reasoning, which is a much higher level of intelligence required for social and professional settings.

Jane: And it shows that even with advanced LLMs, this specific ability to hypothesize those kinds of explanations remains largely unexplored territory in their reasoning capabilities.

Lu: They are providing a framework for automatic evaluation based on NLI judgments rather than just n-gram overlaps, which is a step toward more rigorous testing of these complex linguistic tasks.

Meng: It's important for us to consider that the success metrics they’ve defined—effectiveness and coherence—are what we need to actually build our production guardrails around if we want AI to handle this kind of reasoning reliably.

Tom: So, the gist is they are introducing a task of generating explanations that render contradictory statements compatible, using NLI datasets for scalable evaluation. That’s what this paper is about before we look at the results.

Jane: And it sets up the next part: what did these models actually find when they tried to perform this task?

Conclusion: Tom: So, wrapping up this look at "Explanation Generation for Contradiction Reconciliation with LLMs," the authors, Jason Chan Zhixue Zhao and Robert Gaizauskas from University of Sheffield, found that while models can produce coherent explanations that don't contradict the premise, making those explanations truly effective is still hard.

Jane: That’s the main point: coherence is easier to hit than effectiveness when you’re asking a model to bridge a gap between two opposing ideas. They also noted that performance varies by model size and mode; for example, mid-sized models showed improvement when they were in thinking mode for this specific task.

Lu: The implication for the field is that we need better ways to train these models specifically on generating those reconciliatory explanations, not just general reasoning. It highlights that the training needs to focus on understanding linguistic nuance and implicatures when resolving conflicts.

Meng: For production use, it means if we want AI to handle complex problems where information seems contradictory, we have a specific benchmark—this task of reconciliation generation—that we need to build and test against before trusting them with critical tasks.

Tom: Ultimately, this work motivates further research into this area because generating those explanations successfully is still proving to be a significant challenge for current LLMs. It’s not about detecting errors; it's about how the model reasons around the conflict itself.

Jane: It shifts the focus from just fixing mistakes to understanding how intelligence operates when faced with conflicting information in a way that actually makes sense for real-world applications.

Lu: They also pointed out that current tests are limited, focusing only on English premise-hypothesis pairs and short contexts, suggesting that future work needs to look at larger document sets or multi-hop contradictions.

Meng: That limitation is important because if we want this to be useful for things like legal documents or long scientific papers, the method needs to scale beyond these small examples.

Tom: So, in simple terms: we can now measure how well an AI explains a contradiction by checking if the explanation makes sense logically and doesn't contradict the initial input. That’s what this paper contributes to our understanding of LLM reasoning right now.

Jane: It’s a step toward building systems that can not only say "this is wrong" but also propose a plausible reason why something might seem contradictory.

University of Sheffield, UK

cs.CL

Submitted: 2026-03-24

Updated: 2026-10-15

Comments: Preprint

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 90/100

The gist: The gist The task of reconciling contradictory statements through natural language explanations is an underexplored but key aspect of intelligence in tasks ranging from conversations to scientific

Key concepts

Reconciliatory Explanation Generation (REG): A task where an AI model must write an explanation that resolves the conflict between two contradictory statements. Success requires the explanation to make the premise and hypothesis logically compatible, meaning they no longer contradict each other.
Effectiveness: A measure of success where combining a statement with its generated explanation leads to a logical entailment (ent). In simple terms, it means the explanation successfully bridges the gap between two conflicting ideas so that both can coexist in the same context.
Coherence: A measure ensuring that the generated explanation itself does not contradict the original premise. For an explanation to be coherent, it must be logically consistent with what came before it, preventing the explanation from introducing new conflicts into the initial setup.

Terminology

Large Language Models (LLMs) — 1× in this paper · explained in 8 episodes
Natural Language Inference (NLI) — 1× in this paper · explained in 2 episodes

Summary

The gist The task of reconciling contradictory statements through natural language explanations is an underexplored but key aspect of intelligence in tasks ranging from conversations to scientific discovery.

Introduction and Motivation

A contradiction is often a crucial trigger for reasoning in human cognition, leading people to hypothesize explanations that render conflicting information compatible rather than simply discarding one observation. This ability to reconcile contradictions is a hallmark of intelligence that manifests across everyday social interactions, professional domains, and scientific inquiry. Existing research on how large language models (LLMs) handle information conflicts predominantly focuses on detection and arbitration—deciding which conflicting statement(s) to accept and which to reject. The paper introduces the novel task of reconciliatory explanation generation (REG), where a model must generate natural language explanations that effectively render two apparently contradictory statements mutually compatible.

Task Formulation

The formal task requires a model Mexpl to generate an explanation e such that Mnli(p + e, h) = ent. (effective) and Mnli(p, e)!= con. ∧ Mnli(e, p)!= con. (coherent). The criteria for success are defined as effectiveness—where the premise combined with the explanation entails the hypothesis—and coherence—where the explanation does not contradict the premise. This formulation allows for automatic evaluation by defining success based on standard NLI three-way judgments.

Methodology and Dataset

The REG dataset is constructed from existing NLI datasets, specifically a subset of MultiNLI re-annotated as ChaosNLI (Nie et al., 2020). From this subset, only instances whose most frequent label is “contradiction” are used, resulting in the ChaosNLI-MNLI-C subset consisting of 275 instances. For each instance in ChaosNLIMNLI-C, models are prompted to “generate an explanation that resolves the apparent contradiction between the premise and the hypothesis”, stating requirements for both effectiveness and coherence. Models evaluated include 18 LLMs across various sizes and variants, tested in both non-thinking and thinking modes.

Evaluation Framework

Explanations are evaluated by multiple NLI judge models that initially label the instances as “contradiction”. The success of an explanation e is automatically evaluated by prompting a judge model to predict the NLI label for three relationships: (i) between p + e and h; (ii) between p and e; and (iii) between e and p. The final success score is computed based on these three conditions: (i) Mnli(p + e, h) = ent. (effective); (ii) Mnli(p, e)!= con. ∧ Mnli(e, p)!= con. (coherent); and (iii) Mnli(p, h).

Key Findings

Most models are reasonably capable of generating coherent explanations that do not contradict the premise. However, generating effective explanations remains more difficult for the majority of models. Notably, there is a substantial performance gap between proprietary models (gpt-5-mini and gpt-5.2) and most open-weight models. A key finding is that extended “thinking” does not necessarily improve performance; in fact, thinking models generally underperform their non-thinking counterparts in terms of coherence. Conversely, for mid-sized models (Qwen3-4B, Qwen3-8B and Olmo-3-7B), thinking substantially improves the effectiveness of their generated explanations. Furthermore, a statistically significant but weak negative correlation was found between wcon (the proportion of human annotators labeling a pair as “contradiction”) and the aggregate mean explanation success rate (-0.2257, p < 0.0005).

Limitations and Future Work

The study notes that current experiments are limited to English premise-hypothesis pairs and relatively short contexts, suggesting future research should expand this framework to evaluate document-level or multi-hop contradictions. The authors also state they do not introduce a formal taxonomy for the various strategies employed by models to reconcile contradictory statements, leaving this for future work. Additionally, the task criteria focus specifically on relational properties and do not yet include requirements for explanations to be factually grounded in external evidence or to generate multiple, diverse hypotheses. The results suggest that generating successful explanations remains challenging and that extended “thinking” does not necessarily improve performance.

The paper is 2603.22735. BARE.

Improvements for AI systems

textbfCapabilities in Contradiction Reconciliation Reasoning: Generation of Explanations for Incompatible Statements (REG Task). This improved system can generate natural language explanations that effectively render two apparently contradictory statements mutually compatible, addressing a key gap in LLM reasoning beyond simple statement arbitration.
textbfScalable and Bias-Insulated Evaluation Framework: Automatic Evaluation via NLI Judge Models. The system utilizes a scalable data selection framework that repurposes existing NLI datasets and introduces automatic evaluation metrics based on standard three-way NLI judgments, enabling evaluation at scale without additional human annotations.
textbfContextual Reasoning Enhancement: Leveraging Thinking Modes for Improved Effectiveness. The system can be tuned to utilize the thinking mode of certain models, as this is shown to substantially improve the effectiveness of their generated explanations for mid-sized models, suggesting a mechanism for deeper reasoning.
textbfModel Robustness Assessment: Distinguishing True Reconciliation from Superficial Justification. The system can be trained to evaluate explanations against adversarial examples, as demonstrated by testing explanations that are relevant-but-ineffective, ensuring the judge model is not prone to accepting irrelevant statements.
textbfGuaranteed Judge Consistency: Leveraging Top-Scoring Models for Stable Assessment. For cost management and consistency, the system can be configured to use a single, top-scoring NLI judge model (like Qwen3-32B) as the sole judge for this experiment, providing a stable baseline for comparison.

Abstract

Existing NLP work commonly treats contradictions as errors to be resolved by choosing which statements to accept or discard. Yet a key aspect of human reasoning in social and professional settings is the ability to hypothesize explanations that reconcile contradictions. For example, "Cassie hates coffee" and "She buys coffee everyday" appear contradictory, but are in fact compatible if Cassie was assigned the daily chore of buying coffee for all her coworkers. Despite advances in the reasoning capabilities of large language models (LLMs), their ability to hypothesize such explanations remains largely unexplored. To address this gap, we introduce the task of reconciliatory explanation generation, where models must generate explanations that effectively render contradictory statements compatible. We propose a novel method of reusing natural language inference (NLI) datasets, and introduce quality metrics that enable scalable automatic evaluation. Experiments with 13 LLMs in thinking and non-thinking modes reveal models' limited success in this task, with longer reasoning chains negatively associated with explanation success. As a proof of concept, we further adapt our introduced task and metrics for reinforcement-learning training, finding that trained models exhibit improved accuracy in non-thinking mode on two narrative reasoning benchmarks (True Detective and MuSR). Our findings highlight reconciliatory explanation generation as a challenge in LLM reasoning and motivate further work to investigate its potential for improving model capabilities.