Beyond Input Understanding: Diagnosing Multilingual Mathematical Reasoning with Directed Acyclic Trace Graphs

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "Beyond Input Understanding".

Jane: Here's the gist of the paper. The gist The gap between strong mathematical reasoning performance in English and lower reliability in many low- and medium-resource languages is often explained by a failure to understand non-English problem statements, but this view is incomplete because controlling the model’s reasoning language can substantially reduce accuracy, suggesting that language also affects reasoning execution itself.

Tom: Input-Reasoning Language Settings The study evaluates four input-reasoning language settings to disentangle input-side understanding from trace-side reasoning execution, including the en → x setting where the problem is in English but the reasoning trace is generated in a target language x.

Jane: The analysis shows that "performance is strongly influenced by the language of the reasoning trace, not only by the input language".

Tom: This effect is most severe when "medium- and high-difficulty problems require low-resource reasoning traces".

Paper summary: Tom: So, we’re talking about the paper "Beyond Input Understanding: Diagnosing Multilingual Mathematical Reasoning with Directed Acyclic Trace Graphs." The main idea here is that we usually think the gap between strong English math skills and weaker performance in other languages is because the model just doesn't understand what's written in that language.

Jane: That’s what most people assume, right? They think the problem statement itself is a barrier. But this paper argues that view is incomplete because controlling which language the model uses for its actual reasoning can really hurt accuracy, suggesting language affects how the reasoning happens internally.

Lu: Exactly. They show that even when you give the problem in English, forcing the model to generate its steps in a different target language can substantially reduce performance. That points to something happening during execution rather than just understanding the initial input.

Meng: So, this isn't just about translating the prompt; it’s about how the AI processes those intermediate steps when it switches languages during its work. That makes sense for deployment, because if a model gets confused mid-process, that’s a failure mode we need to catch.

Lalam: I see what they are doing with this "Beyond Input Understanding" title—it shifts the focus from the input side to the trace side of reasoning execution itself. They introduce something called DATG to look at that process structure directly.

Tom: Right, and the core idea behind DATG is mapping those long reasoning traces into something language independent, like mathematical anchors and dependencies. This lets them check if those target-language traces are actually hitting all the necessary mathematical nodes and respecting the required steps.

Jane: So they’re not just looking at whether the final answer is right, but whether the path taken to get there follows a correct mathematical structure, regardless of which language it’s written in. It seems like a really precise way to diagnose execution failure.

Conclusion: Tom: The paper "Beyond Input Understanding: Diagnosing Multilingual Mathematical Reasoning with Directed Acyclic Trace Graphs" suggests that the performance gap isn't just about input understanding; it’s also about failures in how the model executes its reasoning traces when switching languages. They introduce DATG to diagnose this by aligning target-language traces against language independent mathematical anchors and dependencies.

Jane: And their findings across twelve languages on models like Qwen3 show that non-English reasoning often struggles with both anchor coverage and dependency fidelity, especially in low and medium resource settings. This means the failure isn't uniform; it’s specific to the language resources available for tracing.

Lu: What I find most interesting is how they define those metrics—Committed Anchor Recall, Path Monotonic Fidelity, and Harmful Action Rate. These are concrete ways to measure structural integrity rather than just looking at the final score on a benchmark.

Meng: From an engineering standpoint, knowing that low-resource reasoning traces often omit key intermediate facts or break the dependency order gives us a clear target for debugging when we deploy these systems in those environments. It tells us exactly where the execution breaks down.

Lalam: The authors then propose two test-time controls, Loop-Retry and Formula-Retry, which they say consistently improve performance in those low-resource languages by targeting these specific failures exposed by DATG.

Tom: So, the implication is that we need to look beyond just the final answer when evaluating multilingual reasoning. We need to care about the structure and ordering of those generated reasoning traces instead of just focusing on whether they match a perfect English answer.

Jane: It suggests that future evaluation for multilingual systems should focus more on how stable and structurally sound the reasoning process is across different languages, not just the final output accuracy. That’s a big shift in how we think about these models.