Same Answer, New Box: How Much of RLVR's Out-of-Domain Gain Is Answer Format?

arXiv:2610.20639 · cs.LG, cs.CL · Submitted 2026-10-02 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "Same Answer, New Box".

Jane: The gist The gain reported by extractors copied from public code arising from RLVR training disappears under ours,

Tom: First, who's behind it and why it matters.

Title and authors: Tom: We're moving on to how they suggest we improve this measurement. The paper calls these improvements suggestions for future work or for better ways to quantify what we're seeing here.

Jane: It seems they are pushing for a more rigorous way to separate the gain in answers from the gain in format, rather than just reporting a single percentage figure >

Lu: One suggestion is to improve reward-based accuracy reporting by explicitly separating "format gain" from "answer gain." They propose quantifying this by comparing scores generated with extractors that look for specific formatting, like a boxed answer, versus those that read the full text >

Meng: That makes sense if we want to know if the model is actually better at solving the problem or just better at following instructions on where to write it >

Lalam: Another suggestion they have is to implement an extractor-aware scoring mechanism. This would let us report how much of the gain reported by extractors copied from public code arises this way by using re-scoring stored outputs with a second extractor >

Tom: That sounds like a practical way to recover that format gain without having to run all the reinforcement learning training over and over again, which is good for efficiency >

Jane: And they also suggest developing a robust confidence metric for out-of-domain gains. This would assess the reliability of reported accuracy gains by measuring how much those gains disappear when using extractors that look in specific locations >

Lu: That ties back to their finding that eighty-two percent of the gain disappears under their method, which gives us a way to flag when a reported gain might be mostly format >

Meng: I like the idea of flagging gains that are mostly format because it helps us focus on where we need to improve things in the model training itself >

Lalam: They also suggest designing a flexible answer extraction pipeline to adapt to different prompting strategies. This would optimize our output format by dynamically choosing between prompts, since they found the share is seventy-nine percent under the boxed-instruction prompt and eighty-six percent under the plain one >

Tom: Tailoring the reporting based on how you ask the model is a useful thing, especially if we want to see which prompting style leads to better answer quality versus just better formatting >

The paper's summary: Jane: So let's go back to what they actually found about this in the paper called Same Answer, New Box: How Much of RLVR's Out-of-Domain Gain Is Answer Format? They are trying to figure out if the boost from reinforcement learning comes from better reasoning or just better formatting >

Lu: The summary of their main finding is that most of the out-of-domain multiple-choice gain reported by harness extractors disappears under their own method, meaning it's a gain in answer format, not an improvement in answers >

Tom: So to put it plainly, they found that when you look at how much models get better on tasks outside their training range, most of that perceived improvement is actually just the model moving its text into a box where the extractor can find it >

Jane: That means that for many out-of-domain benchmarks, the reported gains are not due to better answers but rather a change in answer format >

Meng: It’s like if you train a car to drive faster, but all you see is that it’s now driving on a different kind of road >

Lalam: They highlight that this happens because base models state most of their answers in text, and the RL models write their answers into boxes >

Tom: So the mechanism is pretty clear: RL training seems to be scaffolding the answer structure for extraction rather than fundamentally improving the underlying knowledge >

Jane: It’s important to note they are looking at this on four released 7B SimpleRL base–RL pairs, and this finding isn't necessarily true for every single model or task > <ref:2610.20639#pg1,on four released 7B SimpleRL base–RL pairs>

Lu: They also found that in in-domain free-form mathematics, where both models already box their answers, an extractor that only reads the boxed answer shows a format gain close to zero >

Tom: So they are saying you can tell the difference between what transfers and what's just structural changes depending on the context of the task >

The paper's improvements: Jane: Now let's talk about those suggestions again. They’re trying to build a system that is more accurate in its reporting. The authors suggest ways to improve how we measure this format gain >

Lu: They suggest the first improvement is separating format gain from answer gain by comparing scores from extractors that look for specific formatting versus those that read the full text, which they call measuring how much of the gain reported by extractors copied from public code arises this way >

Meng: That sounds like a way to quantify it, not just say it's mostly format; we need a number for what we're losing when we only look at boxes >

Lalam: They also suggest an extractor-aware scoring mechanism. This would be doing that by re-scoring stored outputs with a second extractor, which they say recovers the format gain without needing to run RL again >

Tom: That’s smart because it means we don't have to redo all the training just to get a better picture of the format effect >

Jane: They also want a more robust confidence metric for out-of-domain gains. This would check reliability by measuring how much those gains disappear when using extractors that look in specific locations >

Lu: That links back to their finding that eighty-two percent of the gain disappears under their method, so it gives us a statistical measure of how much the reported gain might be noise >

Meng: I see this as a way to build trust in the numbers we see when we look at these benchmarks >

Lalam: Finally, they suggest designing a flexible answer extraction pipeline to adapt to different prompting strategies. This would optimize our output format by dynamically choosing between prompts, since they found the share is seventy-nine percent under the boxed-instruction prompt and eighty-six percent under the plain one >

Tom: So it's about building a tool that helps us report these gains in a way that accounts for how we interact with the model >

Conclusion: Jane: Alright, let’s wrap this up. The main implication of this work is that when we look at RLVR models getting out-of-domain gains, a large part of that credit might actually belong to the program that read the answers >

Lu: So even if the RL process improves reasoning, if it just moves the answer into a box, that credit shifts because the extraction method is what's changing >

Tom: The final thought is to name your extractor behind every reported number and report on how much gain comes from format gain before any RL model exists >

Meng: For deployment at scale, we need to make sure we are measuring what actually matters for real traffic, not just these specific benchmark numbers >

Lalam: And I'm confident that by using this measurement of format gain in papers like Same Answer, New Box: How Much of RLVR's Out-of-Domain Gain Is Answer Format?, we can start separating the structural changes from actual answer quality >

Jane: That’s a clear direction for where this research is going. We have a lot to think about as we look at the next paper on arXiv >

Monash University

cs.LG, cs.CL

Submitted: 2026-10-02

Updated: 2026-10-02

Comments: 36 pages (10 main), 5 figures, 28 tables

Code: https://github.com/openai/simple-evals

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 88/100

The gist: The gist The gain reported by extractors copied from public code arising from RLVR training disappears under ours, indicating that most out-of-domain multiple-choice gains are due to a change in

Key concepts

RLVR
Reinforcement Learning with Verifiable Rewards is a training method where a model learns by receiving rewards based on how well its output matches a desired format or structure, rather than just the correctness of the final answer. This method is credited with achieving gains in specific domains like mathematics.
Format Gain
Format gain refers to the improvement in performance that comes from changing *where* a model places its answer—for example, moving a stated answer into a specific box or text field—rather than improving the actual content of the answer itself. The paper measures how much reported gain is due to this structural change.
Extractor Dependence
The reported performance gain is highly dependent on which 'extractor' (the tool used to read the model's output) is employed. Different extractors yield different scores, suggesting that the perceived benefit of RLVR can be heavily influenced by how strictly or broadly the extraction method interprets the model's response.
Splitting a Reported Gain
This core method involves comparing two scoring methods: one using a harness extractor (which reports the original gain) and another fixed extractor that treats both models uniformly. By calculating the difference between these two scores, researchers can isolate and quantify the 'format gain' component from any reported performance boost.

Summary

The gist The gain reported by extractors copied from public code arising from RLVR training disappears under ours, indicating that most out-of-domain multiple-choice gains are due to a change in answer format rather than better answers

How it works

The paper measures how much of the gain reported by extractors copied from public code arises from a model learning to write its answer where the extractor looks, which is termed format gain rather than an improvement in answers This is done by generating outputs from a base model and its RL model and scoring them twice: with those extractors, and with one of our own that reads a stated answer whether it is in the answer box or in the text The comparison shows that on four released 7B SimpleRL base–RL pairs, 82% of the out-of-domain multiple-choice gain that these extractors report disappears under ours: it is a gain in answer format, not in answers

Splitting a Reported Gain

The core method involves reading the same outputs twice: once with a harness extractor whose gain is the reported one, and once with a fixed extractor that treats both models alike The format gain is defined as D = grep − gfix, where grep is the harness extractor's gain and gfix is the fixed extractor's gain in answers, because omega applies the same rules to both models' outputs This approach allows researchers to tell apart whether a reported gain means what the model learned transfers or if it is merely a change in where it writes its answer

Key Findings on Format Gain

The study finds that Most of the out-of-domain multiple-choice gain is format gain Specifically, on MMLU, MMLU-Pro and ARC-Challenge, 82% of the gain that harness extractors report for four released 7B SimpleRL base–RL pairs is format gain This suggests that RL moves the stated answer into a box The base models state most of their answers in text, and the RL models write most of theirs in box In in-domain free-form mathematics, 2 where both already box their answers, an extractor that reads only boxed shows a format gain close to zero

Dependence on Extractor and Prompt

The reported gain is highly dependent on the extractor used The largest and smallest gains reported by harness extractors for the same pair, benchmark, and prompt differ by a median of 15 points Furthermore, an extractor that also reads letters in the text reports much less format gain The share moves by a few points when the fixed extractor reads more or less strictly, when one pair or benchmark is left out, or when only one of our two prompts is used Independent LLM readers of the whole responses, in place of the fixed extractor, find 77%

Robustness and Limitations

The results are robust because they are measured on released checkpoints with no weights changed, and they do not require re-running RL The paper notes that A reported gain depends on its extractor The study also shows that the scale of the gain matters; for a 32B model trained with another recipe, almost none of the reported gain is format gain Finally, it concludes that Three practices make a format gain visible, and none needs RL to be re-run

Conclusion

The paper contributes this measurement: how much of the out-of-domain multiple-choice gain that harness extractors report for released RLVR checkpoints is format gain The main result is the 82% of Section 4.1, pooled over its 46 comparisons The final conclusion is that "Name the extractor behind every reported number. Report the harness extractor’s extraction rate on the base model: it can be measured before any RL model exists, and in our main result the extractor that reads the most of the base models’ outputs reports the least format gain"

How it works

The paper provides a detailed breakdown of how different factors influence this format share For instance, Leaving out one harness extractor never removes the last boxed rule, which the MATH and Open-Reasoner-Zero extractors share The analysis also shows that the base model’s boxing rate falls as the boxing rate rises across combinations

Key Findings on Format Gain

The study finds that the format gain is not identified for certain comparisons, such as those where the fixed extractor reads no answer in 56 of the base model's 500 outputs and in 7 of the RL model's The analysis also shows that the readers’ reading changes the gain on all items and corrects omega’s fixed-extractor gain by the sample estimate"

Conclusion

The paper concludes that By either reading, then, most of the reported gain is format gain The final takeaway is that When an RLVR model is credited with an out-of-domain gain, part of the credit can belong to the program that read its answers

How it works

The paper provides a detailed breakdown of how different factors influence this format share For instance, Leaving out any one pair moves the share by at most 0.032 The analysis also shows that the prompt matters too, and in the direction the mechanism predicts

Key Findings on Format Gain

The study finds that The fixed extractor itself could be producing the result, so we also had the responses read by readers that share no code with it The analysis also shows that the readers’ format share keeps each comparison’s reported gain on all items and corrects omega’s fixed-extractor gain by the sample estimate

How it works

The paper provides a detailed breakdown of how different factors influence this format share For instance, "Leaving out any one pair moves the share by at most 0.

Improvements for AI systems

  1. Improve reward-based accuracy reporting by explicitly separating format gain from answer gain. The improved system can quantify this by comparing scores generated with extractors that look for specific formatting (like a boxed answer) versus those that read the full text, as shown in Section 2 where the format share is measured.

  2. Implement an extractor-aware scoring mechanism to mitigate reliance on single evaluation methods. The system can be designed to report how much of the gain reported by extractors copied from public code arises this way by using re-scoring stored outputs with a second extractor, which was shown to recover the format gain without re-running RL.

  3. Develop a robust confidence metric for out-of-domain gains. The system can assess the reliability of reported accuracy gains by measuring how much those gains disappear under ours when using extractors that look in specific locations, as evidenced by the finding that 82% of the out-of-domain multiple-choice gain that these extractors report disappears under ours.

  4. Design a flexible answer extraction pipeline to adapt to different prompting strategies. The system can optimize its output format by dynamically choosing between prompts, as the share is 79% under the boxed-instruction prompt and 86% under the plain one, allowing for tailored performance reporting based on user intent.

  5. Integrate a sensitivity analysis framework to identify spurious gains. The system can flag reported gains that are not due to improved reasoning, noting that headline RLVR gains conflate policy improvement with measurement confounds and distinguishing format gain from actual answer quality.

Sources

Related papers