SimCT: Recovering Lost Supervision for Cross-Tokenizer On-Policy Distillation

arXiv:2605.07711 · cs.CL · Submitted 2026-05-08 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: Today's paper: "SimCT: Recovering Lost Supervision for Cross-Tokenizer On-Policy Distillation".

Jane: The gist On-policy distillation (OPD) implicitly assumes that teacher and student predictions are comparable token by token,

Tom: First, who's behind it and why it matters.

Title and authors: Tom: So we're talking about this paper called "SimCT: Recovering Lost Supervision for Cross-Tokenizer On-Policy Distillation," and it’s addressing a problem where you try to transfer knowledge from a big teacher model to a smaller student model, but the tokenizers used by those two models don't even agree on what the words are.

Jane: That sounds really frustrating for anyone trying to get that distilled knowledge working smoothly, especially when you have different AI models that use different vocabulary sets and sequence boundaries. What exactly is this SimCT paper proposing to fix?

Lu: Basically, standard on-policy distillation breaks down because it assumes the teacher and student are looking at the exact same text unit at every single supervised position, but if they tokenize things differently, that assumption fails completely when you try to match tokens directly.

Meng: So, what's the core issue here? Is it just that they use different vocabularies or something more fundamental about how the models see the text?

Tom: It’s both vocabulary mismatch and sequence mismatch. Vocabulary mismatch means they often only share a small part of their vocabularies across different model families, so you can't supervise things that don't overlap.

Jane: And sequence mismatch means even if you have shared tokens, the same piece of text can be broken into totally different sequences by the two tokenizers, so what one model sees as a unit, the other sees as something else entirely.

Lu: SimCT tackles this by moving away from assuming exact token overlap and instead proposing a common supervision space U and scoring maps for both teacher and student predictions over that space.

Meng: How do they define this common space? If we just use shared tokens, we run into the vocabulary problem again, right?

Tom: Exactly. SimCT generalizes the idea by defining a common supervision space USimCT, which includes the overlap of vocabularies and then adds something called A, which are these minimal aligned units.

Jane: Minimal aligned units sound like they’re trying to find the best common ground between what the teacher tokenizer and student tokenizer can actually see at once, even if those pieces look different to them.

Lu: They define these minimal aligned units as the finest boundary-consistent text units that both tokenizers can realize, which could be represented by one or more teacher tokens and also by one or more student tokens.

Tom: And they show that this approach works because the loss function, LSimCT, compares the continuation scores of both models over this new USimCT space. It keeps the basic OPD loss form but changes what it’s comparing against.

Jane: But there’s a key theoretical finding here about why these minimal units are better than just merging them into coarser supervision units.

Title and authors: Lu: There is a proposition, Proposition one that shows that KL divergence between the continuation scores of the student and teacher over these minimal units is greater than or equal to the KL divergence over any coarser supervision units <ref:2605.07711#pg1>.

Tom: So what does that actually mean for us in practice? If we use coarser units, we’re losing signal about where the teacher and student disagree at a finer level.

Jane: It means that if you group those minimal aligned units together into something bigger, you lose the specific teacher-student differences that are visible when looking at those individual minimal units.

Meng: That sounds like they are saying don't simplify things too much if you want to capture the real differences between the models.

Tom: Right, and they suggest that using these finest tokenizer-compatible supervision units is better than using any coarser alternatives for this kind of cross-tokenizer problem.

Jane: This moves us from just hoping for shared tokens to actively constructing a supervision interface that respects how different tokenizers actually process the text.

Lu: The experimental results show they tested SimCT across three model families on mathematical reasoning and code generation benchmarks, and it consistently beat both the SFT method and other representative cross-tokenizer OPD baselines.

Tom: What’s the actual performance gain we’re talking about? Is it just a small tweak, or is it something substantial?

Jane: They reported an average relative gain of six point zero percent over the standard SFT method, which they found with only a modest additional computational cost compared to shared-vocabulary OPD <ref:2605.07711#pg3>.

Meng: Six percent is significant if you’re looking at deployment cost and performance trade-offs on real traffic, but it’s not a massive jump that would make it instantly deployable without thinking about inference speed.

Tom: The paper suggests that exact shared-token supervision is too conservative when tokenizers mismatch, and SimCT recovers useful supervision from text continuations that are comparable but not represented as identical tokens.

Jane: It’s like the teacher says "this is the next word," and the student says "this is a different word," but SimCT finds a way to compare them because both models can see that this segment of text is meaningful.

Lu: Qualitative case studies support this idea, showing how SimCT handles things like number theory problems by precisely computing results, avoiding errors that baselines make in final steps.

Tom: And with code generation, it’s not just about getting the right syntax; SimCT seems to capture algorithmic efficiency too.

Jane: It lets the student model infer the expected function interface correctly from the problem context, which means it generates solutions that are both syntactically correct and algorithmically sound.

Meng: That’s important because for production code, generating an efficient solution is often just as critical as getting a result that doesn't crash.

Title and authors: Tom: It seems to be highly effective at transferring not just syntax but also the underlying logic from the teacher model when tokenization gets messy.

Jane: So, what does this mean for the broader application of knowledge distillation across different AI architectures?

Lu: The implication is that changing the supervision space itself, rather than just changing how you train with OPD, is an effective way to handle these tokenizer mismatches in cross-tokenizer settings.

Tom: It suggests we should focus on constructing better alignment mechanisms for these models instead of just sticking to the standard training paradigm.

Jane: So, overall, the paper "SimCT: Recovering Lost Supervision for Cross-Tokenizer On-Policy Distillation" shows that by enlarging the supervision space with minimal aligned units, you can recover a lot of teacher signal even when tokenizers don't match up perfectly.

Meng: From a deployment standpoint, it’s good that it doesn't add trainable projection modules or any other complexity; it just fits into the standard OPD training pipeline.

Lu: The authors also pointed out that this method relies on mapping tokenizer mismatches to minimal aligned units, which is natural for standard BPE-style tokenizers, but they acknowledge it might need adaptation for tokenizers with much different segmentation behavior.

Tom: And they did note that they didn't exhaustively study things like character or byte-level normalization as alternatives, which is a limitation we should keep in mind.

Jane: Exactly. The paper focuses on the white-box OPD setting where teacher next-token distributions are available, which is helpful for understanding the mechanism itself.

Meng: So, while it’s good for research and getting better results, it’s important to remember that this work is centered on mathematical reasoning and code generation benchmarks right now.

Tom: Right. But the results show a consistent improvement over SFT and existing cross-tokenizer baselines across those specific challenging domains.

Jane: So, to wrap up this discussion of "SimCT: Recovering Lost Supervision for Cross-Tokenizer On-Policy Distillation," the main point is that you can recover useful teacher feedback by comparing models over these minimal aligned units instead of just relying on exact token matching.

Lu: This paper gives us a solid idea for how to design a supervision interface that respects the underlying text structure across different tokenizers.

Meng: It’s efficient enough to be considered for deployment, with only a modest computational overhead compared to the other shared-vocabulary approaches.

Tom: It shows that adapting the supervision space is more effective than trying to force exact token overlap when models use different tokenization schemes.

Jane: So, we've looked at how SimCT uses minimal aligned units to restore lost supervision in cross-tokenizer distillation, and it seems like a promising direction for improving knowledge transfer between different AI models.

The paper's summary: Tom: So, to recap what we just heard, SimCT is this new method that tries to fix on-policy distillation when the teacher and student AI models use different ways of chopping up text—different tokenizers.

Jane: Right, and the big idea is they realized that if those tokenizers don't agree on what a word or a piece of text unit is, then you can't just match tokens one-to-one anymore.

Lu: Exactly. Standard distillation breaks because it assumes the teacher and student are looking at the exact same thing token by token, which fails when their vocabularies don't overlap perfectly.

Meng: So they’re basically trying to find a way to supervise them over something that both models can actually see, even if they see it differently.

Tom: Yeah, and what they propose is this common supervision space USimCT that includes the overlap of their vocabularies plus these minimal aligned units.

Jane: That’s the core mechanic. These minimal aligned units are defined as the finest text segments that both tokenizers can represent, even if one model sees it as a bunch of tokens and the other sees it as something else.

Lu: The theoretical analysis shows that this approach is sound because when you look at those minimal units, the teacher and student have a clear distinction in their next-token predictions.

Tom: And they prove that if you take those minimal units and group them into something coarser, like merging them together, you lose that important distinction between the teacher's preference and the student's preference within that span.

Jane: It’s like if you group all those little pieces of text together, you smooth out the differences between what one model thinks is important and what the other model thinks is important for that specific segment.

Meng: So, to deploy this on a real system, it means we don't need to force a single tokenizer onto both models; we just define this shared space and let them compare their scores there.

Tom: And the results show that SimCT actually performs better than the standard SFT distillation baseline, showing about a six percent relative gain across math and code tasks.

Jane: That means for someone listening, it suggests that you don't need perfect token-for-token alignment to get good knowledge transfer from a big model to a smaller one.

Lu: The implication is that changing the supervision space itself, instead of just tweaking the training algorithm, is an effective way to handle cross-tokenizer distillation problems.

Tom: It’s efficient too; it doesn't require any extra trainable modules or projection layers, which keeps the overhead pretty low compared to other shared-vocabulary methods.

Jane: So this could change how we approach transferring knowledge between different AI architectures that were trained with completely different tokenization schemes.

Lu: And while they focus on math and code generation for now, the mechanism itself is flexible enough to handle various segmentation behaviors in BPE-style tokenizers.

Meng: But they did flag a limitation: this method relies on mapping mismatches to these minimal aligned units, which might need some adjustments for tokenizers that segment text in a way that's very different from standard BPE.

Tom: And they also didn't fully explore things like character or byte-level normalization as alternatives, so we still have room there for future work.

Jane: So while SimCT gives us a solid way to recover teacher signal when tokenizers mismatch, it’s important to keep an eye on those other normalization techniques for more robust cross-model training down the line.

The paper's improvements: Tom: So, if we’re looking at how SimCT actually improves things, they aren't just changing the loss function; they're fundamentally changing what we use to supervise those models.

Jane: Right, so instead of just comparing tokens directly, SimCT builds a whole new way to look at the data that both tokenizers can agree on.

Lu: The main improvement is this construction of USimCT, which combines the shared vocab overlap with those minimal aligned units we talked about before.

Tom: What’s interesting is how they prove that these minimal units are superior because they preserve the fine-grained differences between teacher and student preferences.

Jane: So what does that mean for us? It means we can capture those subtle things, like a specific logical step in math or a tricky function interface in code generation.

Meng: For production deployment, that precision is key; it means the distilled model isn't just guessing; it’s learning the right way to do things at a deeper level.

Tom: And they showed that coarsening those units—making them bigger—actually removes that teacher-student signal we care about, which is a huge theoretical win for their method.

Jane: So, what does this mean for someone just listening? It means the AI transfer is more accurate and less likely to make mistakes on complex tasks because it’s looking at the same high-quality signal.

Lu: The experimental results confirm this; they saw a consistent gain over other cross-tokenizer baselines across math and code benchmarks.

Tom: And that gain of about six percent over standard SFT is solid, especially considering how little extra computational cost SimCT adds compared to just using shared-vocabulary distillation.

Jane: It suggests that the problem isn't necessarily needing a whole new training setup; it’s about making the supervision space smarter and more flexible for different model tokenizers.

Meng: So, if I were shipping this, I wouldn't need to overhaul my entire inference pipeline just because of tokenizer differences; I could just use this method to fine-tune the distilled model.

Lu: That’s exactly it; it’s about adapting the supervision interface to respect how the different tokenizers actually process the text, not forcing a one-size-fits-all solution.

Tom: It points towards a general principle: for cross-tokenizer distillation, modifying the supervision space is often a more effective path than just changing how you structure your on-policy training.

Jane: So, while they focused on math and code generation, it opens up possibilities for applying this to any area where you're distilling knowledge between models with different tokenization styles.

Conclusion: Tom: So, to wrap up this SimCT paper, it’s basically about using minimal aligned units to fix on-policy distillation when models use different tokenizers.

Jane: Right, and the main implication is that you can get much better knowledge transfer even when the teacher and student AI models have totally different vocabularies.

Lu: They show that focusing on those fine-grained text units, instead of coarser ones, actually preserves the important teacher-student signal we need for accurate reasoning.

Meng: For deployment, that means we can be confident that our distilled model is learning the right logic and not just surface patterns because it’s getting better supervision.

Tom: The numbers back this up; they saw about a six percent relative gain over standard SFT methods on math and code benchmarks, which is solid.

Jane: It really changes how we think about cross-model training, suggesting that modifying the supervision space itself is a more powerful tool than just changing the distillation training process.

Lu: The authors are careful to point out their limitation: this method relies on mapping token mismatches to these minimal aligned units, which might need some adaptation for tokenizers with very unusual segmentation behaviors.

Meng: And they didn't cover alternatives like character or byte-level normalization, so that’s a piece of the puzzle we still need to consider for real-world robustness.

Tom: So, the paper on "SimCT: Recovering Lost Supervision for Cross-Tokenizer On-Policy Distillation" gives us a concrete way to recover useful teacher feedback without needing perfect token overlap.

Jane: It’s a solid direction for improving how we transfer knowledge between AI models that have different underlying text representations.

Lu: It’s the idea of building this supervision space smartly, rather than just assuming everything lines up perfectly on a word-for-word basis across vocabularies.

Meng: For me, it’s about making sure the model doesn't break when we try to deploy it at scale because we know the distillation process actually learned something meaningful.

Tom: Yeah, so that’s the gist of SimCT: expanding the supervision space to recover lost signals when tokenizers mismatch.

Jane: We’ve looked at how these minimal aligned units help restore precision in things like code generation and mathematical reasoning.

University of Science and Technology of China

cs.CL

Submitted: 2026-05-08

Updated: 2026-10-15

Comments: Accepted by NeurIPS 2026

Code: https://github.com/sunjie279/SimCT-

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 90/100

The gist: The gist On-policy distillation (OPD) implicitly assumes that teacher and student predictions are comparable token by token, an assumption that fails whenever the two models tokenize the same text

Key concepts

On-policy Distillation (OPD)
A technique used to transfer knowledge from a large 'teacher' model to a smaller 'student' model during training. It usually assumes both models use the exact same tokens for comparison, which is problematic when tokenizers differ.
Cross-Tokenizer Mismatch
The problem where two different language models use different tokenization methods. This causes issues because the teacher and student might segment the same piece of text into completely different sequences of tokens, making direct comparison impossible.
Minimal Aligned Units (A)
These are the finest possible text segments that both tokenizers can represent consistently. SimCT identifies these units, which serve as a common supervision space. By focusing on these fine units instead of coarser ones, the method preserves the most useful teacher-student distinctions.
SimCT Loss (LSimCT)
The specific loss function used by SimCT to compare teacher and student predictions over the expanded supervision space. It measures similarity between the continuation scores of both models mapped onto this common, finer set of aligned units.

Terminology

Summary

The gist On-policy distillation (OPD) implicitly assumes that teacher and student predictions are comparable token by token, an assumption that fails whenever the two models tokenize the same text differently, which SimCT addresses by enlarging the supervision space to recover lost signals

Problem Setup

The core issue in cross-tokenizer OPD is that standard OPD breaks its basic assumption when teacher and student models use different tokenizers, leading to two main obstacles: vocabulary mismatch and sequence mismatch. Vocabulary mismatch occurs because modern LLM tokenizers share only a limited portion of their vocabularies across model families, restricting supervision to tokens that appear in both vocabularies and sequence mismatch arises because the same text may be segmented into different token sequences, meaning the teacher’s next-token distribution at a student-generated prefix may not supervise the same text unit as the student’s next-token distribution.

Generalized OPD over a Common Supervision Space

To handle these mismatches, cross-tokenizer OPD is generalized by replacing the shared-vocabulary assumption with a shared set of prediction units, illustrated by introducing a common supervision space U and scoring maps ΠT and ΠS. This allows for the comparison of next-token distributions over this common space U, where the loss is defined as LgOPD = D(ΠS pS · x<t, ΠT pT · x<t). The central design choice is how to construct U so that it preserves useful teacher supervision while remaining compatible with student predictions.

SimCT: Recovering Supervision via Minimal Aligned Units

Simple Cross-Tokenizer OPD (SimCT) expands the common supervision space beyond exact token overlap by recovering teacher supervision whenever teacher and student predictions refer to the same underlying text, even if the two tokenizers realize that text with different token boundaries. SimCT constructs its supervision space as USimCT = (VT ∩ VS) ∪ A, where A is the set of minimal aligned units. These minimal aligned units are defined as the finest boundary-consistent text units realizable by both tokenizers, which can be represented by one or more teacher tokens and also by one or more student tokens. The SimCT loss is LSimCT = D(qSimCT S · x<t, qSimCT T · x<t).

Properties of the SimCT Supervision Space

Theoretical analysis shows that minimal aligned units provide the finest comparable text units jointly expressible by the teacher and student tokenizers, while merging these units into coarser supervision units can remove teacher–student distinctions that are visible at the minimal-unit level. Specifically, Proposition 1 demonstrates that KLqmin S∥ qmin T ≥ KLqC S∥ qC T, showing that coarsening removes exactly the within-unit teacher–student discrepancy. This supports using the finest tokenizer-compatible supervision units rather than coarser alternatives.

Experimental Results and Conclusion

Across three model families on mathematical reasoning and code-generation benchmarks, SimCT consistently improves over SFT and representative cross-tokenizer OPD baselines, achieving an average relative gain of 6.0% over SFT with modest additional computational cost relative to shared-vocabulary OPD. The results show that exact shared-token supervision is too conservative under tokenizer mismatch, while SimCT recovers supervision from text continuations that are comparable but not represented as identical tokens. Overall, the findings suggest that changing the supervision space rather than the OPD training paradigm is an effective path for cross-tokenizer OPD.

Case Studies

Qualitative case studies illustrate how different supervision interfaces affect cross-tokenizer OPD behavior at the local level. For instance, in a number theory example, SimCT precisely computes 10000 = 7 × 1428 + 4, while baselines make arithmetic errors in the final modular division step. In code generation, SimCT generates an efficient O(log X) iterative solution that correctly infers the expected function interface from the problem context. These examples highlight that span-level contrastive training transfers not only syntactic correctness but also algorithmic efficiency and proper I/O handling from the teacher model.

Computational Resources

All experiments were conducted on a single server equipped with dual AMD EPYC 9K84 CPUs, 384 logical CPU cores, 2.2TB system memory, and 8× NVIDIA H20 GPUs with approximately 96GB memory per GPU. The total cost is approximately 14.8 days on a single 8× H20 server. SimCT operates at the distillation-interface level, introducing no trainable projection module and keeping the standard OPD training pipeline unchanged.

How it works

  1. The student first generates a full on-policy response y.

  2. The SimCT supervision space is constructed as USimCT = (VT ∩ VS) ∪ A(y), where A(y) are the minimal aligned units.

  3. Continuation scores sM(u x<t) are computed for both models over USimCT using Eq. (7).

  4. The final SimCT loss is LSimCT = D(qSimCT S · x<t, qSimCT T · x<t).

Limitations

The evaluation centers on math and code reasoning with three teacher–student pairs spanning two teacher families and heterogeneous tokenizers; extending the study to additional domains, scales, and tokenizer families is left to future work. Methodologically, SimCT relies on mapping tokenizer mismatches to minimal aligned units, which is natural for standard BPE-style tokenizers and may require adaptation for tokenizers with substantially different segmentation behavior. Additionally, the authors do not exhaustively study alternatives such as no normalization or character/byte-level normalization. Finally, they target the white-box OPD setting in which teacher next-token distributions are available.

How it works

The paper introduces SimpleOPD as a conservative shared-vocabulary baseline and proposes SimCT, which expands the common supervision space with minimal aligned units while preserving the standard OPD objective. The theoretical analysis characterizes these units as the finest jointly expressible supervision interface and shows that coarsening them can erase useful teacher–student distinctions.

How it works

The SimCT algorithm involves tokenizing the same rollout with both tokenizers, extracting minimal aligned units A(y), defining USimCT, and then computing continuation scores sM(u x<t) via Eq. (7) before mapping both models to this space using Eq. (8).

How it works

The paper demonstrates that coarsening removes exactly the within-span KL signal that distinguishes teacher and student preferences inside each merged span. This supports using SimCT’s minimal aligned units instead of merging them into coarser units.

Improvements for AI systems

  1. Bold header: Supervised on heterogeneous tokenizers through minimal aligned units

SimCT restores this signal by enlarging the supervision space: alongside shared tokens, SimCT compares teacher and student over short multi-token continuations that both tokenizers can realize. This allows the system to perform on-policy distillation effectively across models with different vocabularies without losing useful teacher feedback.

  1. Bold header: Robustness against tokenizer mismatch

The system achieves consistent gains over shared-vocabulary OPD and representative cross-tokenizer baselines, showing that SimCT is not tied to a particular teacher, student, or task type, but provides a robust supervision interface for cross-tokenizer OPD. This means the distilled knowledge transfer remains effective even when teacher and student models use different tokenizers.

  1. Bold header: Preserving local reasoning precision

By recovering supervision from minimal aligned units, SimCT prevents the erasure of fine-grained teacher–student distinctions, such as those crucial in mathematical reasoning or code generation, which simultaneously improves over SimpleOPD and avoids errors like an arithmetic error in the final modular division step.

  1. Bold header: Accurate handling of implicit API contracts

SimCT allows for the generation of code with correct function signatures because it correctly infers the expected function interface from the problem context, a capability where baselines fail, such as when they raise a TypeError due to the unexpected second argument.

  1. Bold header: Efficient and lightweight distillation

The method is highly efficient as it requires no additional parameters, no learned projections, and no transport computation, meaning it can be deployed with a modest additional computational cost relative to shared-vocabulary OPD while achieving superior performance.

Abstract

On-policy distillation (OPD) is a standard tool for transferring teacher behavior to a smaller student, but it implicitly assumes that teacher and student predictions are comparable token by token, an assumption that fails whenever the two models tokenize the same text differently. Under heterogeneous tokenizers, exact shared-token matching silently discards a large fraction of the teacher signal at precisely the positions where vocabularies disagree. We propose ple ross- okenizer OPD (SimCT), which restores this signal by enlarging the supervision space: alongside shared tokens, SimCT compares teacher and student over short multi-token continuations that both tokenizers can realize, leaving the OPD loss form itself unchanged. We show that these units are the finest jointly tokenizable supervision interface, and that coarser alternatives remove teacher-student distinctions that are useful for on-policy learning. Across three heterogeneous teacher-student pairs on mathematical reasoning and code-generation benchmarks, SimCT shows consistent gains over shared-vocabulary OPD and representative cross-tokenizer baselines, with ablations confirming that the improvements come from recovering supervision discarded by exact shared-token matching. Code is available at https://github.com/sunjie279/SimCT-.

Sources

Related papers