What MLLMs Learn about When they Learn about Multimodal Reasoning

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "What MLLMs Learn about When they Learn about Multimodal Reasoning".

Jane: Here's the gist of the paper. Evaluation of multimodal reasoning models is typically reduced to a single accuracy score, implicitly treating reasoning as a unitary capability, as introduced in MATHLENS, which decomposes performance into perception, reasoning, and multimodal-specific components The gist is that common training strategies induce systematically different capability profiles that are invisible under aggregate accuracy.

Tom: How it works MATHLENS is a benchmark of textbook-style geometry problems that exposes the assumption of unitary reasoning by operationally decomposing performance into perception, reasoning, and multimodal-specific components.

Jane: Each problem is derived from a symbolic specification and accompanied by visual diagrams, text-only variants, multimodal questions, and targeted perceptual probes.

Tom: The decomposition allows for controlled measurement of each component.

Paper summary: Tom: So we're talking about this paper called "What MLLMs Learn about When they Learn about Multimodal Reasoning." Essentially, they argue that when we just look at one overall accuracy score for a multimodal model, we’re missing something big.

Jane: Right, so the core idea here is that this single score treats reasoning like it's one single thing, but it actually breaks down into several parts: perception and reasoning and then something specific to handling different modes.

Lu: The authors introduce a benchmark called MathLens to show this assumption is flawed by breaking performance down into those three measurable pieces. They do this by taking textbook geometry problems and creating different versions of them, like having visual diagrams, just text descriptions, multimodal questions, and even specific probes for looking at visual details.

Meng: So it’s not just one test; they’re building a whole system to see what part of the model is failing—is it seeing the picture right? Is it solving the math correctly when you only have words? Or is it failing because it can't connect them properly?

Lalam: I see, so they're using this decomposition to show that common training methods actually lead to different internal profiles for these capabilities that you just can't see when you look at the final aggregate accuracy.

Tom: Exactly. The paper claims that training strategies induce systematically different capability profiles, and those differences are hidden by the simple accuracy number. It’s about how the model is learning differently under those same conditions.

Jane: And they point to some specific findings about what kind of training boosts which part of the model, and this is where it gets really interesting for us to hear.

Lu: They find that reinforcement learning primarily helps with perceptual grounding and making the models more robust when the diagrams change, but textual supervised finetuning actually brings gains through reflective reasoning skills.

Meng: That’s a clear split in how different training objectives affect those components; RL is better for seeing what’s there visually and staying stable when things shift, while text SFT improves the model's ability to reason by thinking about it more deeply.

Lalam: And they also observe that as perception and reasoning improve, a growing chunk of the remaining errors falls into this category called multimodal-specific residuals that you can’t measure with the standard aggregate score.

Tom: That suggests that apparent progress in multimodal reasoning isn't uniform advancement across the board; it’s more about shifting balances among these different subskills.

Jane: So what does this actually mean for us when we're trying to train or evaluate these models? It suggests we need to look beyond that single accuracy metric if we want a real picture of what the AI is doing.

Lu: The implication is that common training strategies aren't universally good; they push the model toward specific strengths, and understanding those shifts helps us design better training objectives for different tasks.

Meng: From a deployment standpoint, it means we can’t just chase a high overall score; we need to know if the model is strong in perception or reasoning specifically for the kind of multimodal inputs we’ll actually be hitting in production.

Lalam: I'm pretty sure they are quite confident in this analysis, especially since they have this detailed decomposition of errors across different training setups.

Tom: The authors show that when you combine things like MM SFT plus RL, the gains shift; for instance, textual SFT mainly helps with reasoning, but multimodal RL applied afterward yields bigger gains by shifting its effect to perception and those multimodal-specific residuals.

Jane: That’s a concrete way to see how different parts of the training pipeline are contributing to different kinds of skill development.

Lu: They also noted that perception gains aren't the same everywhere; for example, angle-related relations improve with multimodal reinforcement learning, but polygon-level relations improve mostly when you include textual supervised finetuning.

Meng: That tells me we need to tailor our training based on the specific geometric skill we want to develop, instead of just using one generic strategy for everything.

Lalam: It seems like they’ve given us a really useful map of where each training method lands in terms of capability improvement.

Tom: So, if you look at the title and authors again, this whole study is about exposing the assumption that reasoning is unitary by showing how it actually operates in pieces when it comes to multimodal tasks.

Jane: It’s a really important distinction for understanding what these large models are actually capable of when they process pictures and text together.

Lu: The authors are pushing for evaluation methods that can measure these components separately, because right now, skill-specific testing without strict controls doesn't give us enough diagnostic value.

Meng: For scaling up, knowing which component is weak tells us exactly where to invest our compute budget for training adjustments or fine-tuning.

Lalam: I think the biggest shift here is moving from a single number to a multi-axial evaluation framework that forces developers to consider perception, reasoning, and those residual errors separately.

Tom: So the paper suggests we need to stop looking at just one number and start looking at how the training method is balancing those different underlying skills.

Jane: It really changes how we think about model progress from just getting a higher score to understanding the mechanics of what’s actually improving inside the model.

Conclusion: Tom: So, we're wrapping up this look at "What MLLMs Learn about When they Learn about Multimodal Reasoning."

Jane: Basically, the paper says that when we just look at one big accuracy score for a multimodal model, we're missing how it actually learns to reason across different kinds of inputs.

Tom: Right. The authors introduce this benchmark called MathLens. They break down performance into perception, reasoning, and these multimodal-specific components.

Jane: It’s like they take geometry problems—text with diagrams, just text descriptions, and visual questions—and test the model on each piece separately to see what's going wrong.

Tom: Exactly. And the key finding is that common training methods actually push models toward different capability profiles that you can't see under a single aggregate score.

Jane: It suggests that apparent progress in multimodal reasoning isn't one uniform step forward; it’s really about shifting balances among those different subskills.

Tom: So, if we look at the title and the authors, they are essentially arguing that we need to move beyond just checking one number for these models.

Jane: It means that how a model is trained actually dictates which skills—seeing the picture versus doing the math—get boosted most effectively.

Tom: That’s what I find interesting because it explains why some training setups work better than others for specific kinds of multimodal tasks.

Jane: And this points toward a future where we can evaluate these models by looking at those different skill components, not just the final score.