CoTS: Competitive Token Selection for Latent Reasoning in Large Language Models

arXiv:2610.20642 · cs.CL, cs.AI · Submitted 2026-10-08 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: Today's paper: "CoTS: Competitive Token Selection for Latent Reasoning in Large Language Models".

Jane: The gist The Competitive Token Selection (CoTS) framework reframes latent reasoning from a binary mode selection into continuous token-level allocation using top-two probability competition to dynamically balance discrete commitment and…

Tom: First, who's behind it and why it matters.

Title and authors: Tom: So, CoTS basically takes that abstract idea of latent reasoning—that hidden thinking process—and treats it like a constant balancing act between being sure and being curious at every single step using competition between candidate tokens.

Jane: That sounds really abstract, so if I'm driving and I suddenly feel like I need to be totally committed to my lane, but then I also want to check the side mirrors just in case there’s a hazard, does this paper mean the AI can do that balancing act automatically?

Lu: Exactly. The core idea is moving away from just picking one path and instead using a top-two competition—the two most likely tokens—to dynamically allocate weight between that single choice and exploring other possibilities through interpolation.

Meng: So, if I'm deploying this, what does that look like operationally? Does it just mean more math happening at inference time?

Tom: It means we get a continuous allocation of information instead of a hard binary switch. They use the ratio between the two most probable candidates to decide how much to blend the sampled token embedding with the softer candidate mixture for the next step.

Jane: So, when one candidate is way more likely than another, that ratio goes down, and you stick really close to that first choice. But if they are pretty close in probability—if the competition is tight—then you let those soft candidates have a bigger influence on what the model thinks next.

Lu: That’s the mechanism for controlling commitment versus exploration. They actually introduce this stability thing too, where they guarantee that at least half of your weight is always anchored to that original sampled token embedding.

Meng: Anchoring at half? Why not more? I mean, if you're exploring, don't you want the soft part to dominate more when the uncertainty is high?

Lalam: The authors argue that this anchoring prevents representation drift. They show that even during this competitive selection process, the distance between your next input and your initial sampled token embedding stays bounded by half of it.

Tom: That’s a solid engineering guarantee for stability. And they have this phase lock mechanism, which uses binary control flags to switch off the reasoning stage and start outputting the final answer once it thinks it's done, while protecting those critical math operators from getting messed up during that transition.

Jane: It sounds like they’re making a very concrete link between the model’s internal uncertainty—how much it’s competing—and how much commitment it shows to a specific path.

Lu: They show this works across six different reasoning benchmarks, and they get superior performance on models like Qwen3 point five-9B and Gemma-four-12B-IT compared to standard CoT and other training-free methods.

Meng: So, the results aren't just theoretical; they hold up when you actually test it against real reasoning tasks, which is what I need to see before I even think about putting this on a production pipeline.

Tom: It’s not just small tweaks; it’s consistently better performance across different model families. They show that the boundary between committing and exploring doesn't have to be fixed; it can evolve continuously based on the model's own uncertainty at each step.

Jane: That implies we might stop trying to manually set those commitment thresholds and let the AI manage that balance itself based on how uncertain it is right now.

Lu: It shifts the tuning effort from figuring out where to draw a line to just optimizing how you set up that initial token competition.

Meng: If this works without needing massive amounts of labeled data just to define those commitment boundaries, that really changes the cost structure for us in terms of data preparation.

The paper's summary: Tom: So, CoTS isn't just about getting better numbers; they are fixing how we think about that "commitment" versus "exploration" part of latent reasoning at a fundamental level using this top-two competition.

Jane: So, what’s the actual mechanism they propose to make that balance continuous instead of binary? I mean, how does the competition between two tokens actually translate into a smooth adjustment for the next step?

Lu: They use this specific formula where the mixing weight alpha is calculated directly from the ratio of probabilities between those top two candidates. When one candidate dominates, alpha approaches zero, and you stay close to that sampled token embedding.

Meng: That means it’s self-regulating based on confidence—if the model is pretty sure about one direction, it commits heavily to it without needing a manual setting.

Tom: Exactly. And they have this phase lock mechanism built in for final answers, which uses binary control flags alpha t equals r t minus d t alpha t, to switch the model from reasoning mode to output mode immediately once an answer is emitted while protecting those math operators and formatting tokens.

Jane: That sounds like a practical solution for production; making sure the model doesn't get confused during that final output stage where things need to be perfectly formatted.

Lu: They also built in this bounding mechanism for stability, where the weight assigned to the sampled token is wt of yt equals one minus alpha t plus alpha t times the probability of yt being in our candidate set St at that moment.

Meng: So even when it’s doing this soft exploration, they guarantee that at least half of your representation weight is always explicitly anchored back to what you initially sampled.

Tom: That prevents the model from drifting too far away from its most likely starting point, which is a big win for keeping the reasoning trajectory stable during those complex steps.

Jane: It means instead of just having a static way to blend information, the model dynamically adjusts how much it leans on that soft exploration versus sticking to its initial choice based on the local competition.

Meng: If this works across different model families, that’s good news because we don't have to rebuild our whole reasoning pipeline just to get that level of internal control.

Tom: The performance gains on Qwen3 point five-9B and Gemma-four-12B-IT show they aren't just adding complexity; they are genuinely improving the model’s ability to reason across different problem types <ref:2610.20642#pg3>.

Jane: So, to sum up the improvements: CoTS takes that fuzzy concept of latent reasoning and turns it into a quantifiable, continuous allocation based on token competition, leading to better performance while keeping the system stable.

Lu: The paper shows this framework can evolve continuously with the model’s own uncertainty at each step without needing external tuning of those switching thresholds.

Meng: It suggests that we could potentially make our reasoning processes much more nuanced without needing massive amounts of labeled data just to define every single commitment boundary manually.

The paper's improvements: Tom: So, to wrap up, CoTS is this training-free method that uses competition between the top two tokens to continuously balance discrete commitment and soft exploration in latent reasoning.

Jane: That means we’re moving away from fixed switches toward a system where the AI decides how much it should explore based on its own local confidence at every single step of complex thinking.

Lu: Exactly, and they showed that this competitive token selection method consistently improves reasoning across different models and scales, achieving high accuracy on things like Qwen3 point five-9B and Gemma-four-12B-IT <ref:2610.20642#pg3>.

Meng: The implication for deployment is that we might stop needing those brittle manual switching thresholds and just rely on the inherent competition between candidates to guide the model’s behavior during inference.

Tom: And they did introduce a phase lock for final answers, which is a nice engineering detail because it protects the math operators from getting messed up when the model commits to an output.

Jane: It sounds like this framework offers a way for AI to be both deeply focused and curious at the same time, depending on what’s happening in the reasoning process right then.

Lu: The stability guarantee they put in place, ensuring half of your weight stays anchored to that sampled token embedding, is a key part of why this works reliably under those competition conditions.

Meng: It sounds like a solid mechanism for getting better performance without needing massive retraining cycles just to get that fine-grained control over how the model behaves internally.

Lalam: I’m confident in my assessment because I’ve seen the literature, and CoTS directly addresses the problem of balancing commitment and exploration at the token level using top-two probability competition.

Tom: It really redefines how we think about latent intervention, moving it away from a binary decision toward a continuous allocation controlled by those relative probabilities.

Jane: It suggests that instead of fixing where the line is between being committed to a path and exploring alternatives, that boundary can evolve naturally based on the model's own uncertainty at each step.

Meng: For us shipping models, this means potentially making our reasoning processes much more nuanced without needing massive amounts of labeled data just to define every single commitment boundary manually.

Tom: So, that’s our rundown on CoTS: it’s a training-free framework using top-two probability competition to dynamically balance discrete commitment and soft exploration, leading to better results across models.

Jane: It really shows that the boundary between being committed and exploring isn't a fixed thing; it can evolve naturally based on how uncertain or confident the model is at each step.

Meng: I’m still thinking about the practical side—how much computational overhead does this dynamic weighting actually add to our inference time compared to just running standard CoT?

Tom: The authors address the stability by bounding the representation drift, ensuring that at least half of your weight stays anchored to the sampled token embedding, which is a key engineering detail for keeping things stable.

Jane: That sounds like a good safety net for production environments, making sure you don’t suddenly jump into a completely unrelated thought because of some soft exploration.

Lu: It shifts the tuning effort from figuring out where to draw the line to just optimizing how you set up that initial token competition, which is much cleaner.

Conclusion: Tom: So we're wrapping up on "CoTS: Competitive Token Selection for Latent Reasoning in Large Language Models." Basically, this training-free method uses competition between the top two tokens to dynamically balance discrete commitment and soft exploration in latent reasoning.

Tom: And they did introduce a phase lock for final answers, which is a nice engineering detail because it protects the math operators from getting messed up when the model commits to an output.

Jane: It sounds like this framework offers a way for AI to be both deeply focused and curious at the same time, depending on what’s happening in the reasoning process right then.

Meng: It sounds like a solid mechanism for getting better performance without needing massive retraining cycles just to get that fine-grained control over how the model behaves internally.

Tom: It really redefines how we think about latent intervention, moving it away from a binary decision toward a continuous allocation controlled by those relative probabilities.

Lu: That continuous adaptation based on local uncertainty is really what makes this framework powerful for representation learning because it mirrors how human thinking changes when we start getting more or less sure about a problem.

Lalam: Overall, CoTS is about using the math of probability competition to solve the problem of balancing commitment and exploration in latent reasoning more effectively than existing methods.

Jane: It really shows that the boundary between being committed and exploring isn't a fixed thing; it can evolve naturally based on how uncertain or confident the model is at each step.

Lu: The paper calls this continuous allocation strategy competitive token selection, which is how they manage that balance without needing externally tuned mixing coefficients for every single step.

Meng: I’m still thinking about the practical side—how much computational overhead does this dynamic weighting actually add to our inference time compared to just running standard CoT?

Jane: That sounds like a good safety net for production environments, making sure you don’t suddenly jump into a completely unrelated thought because of some soft exploration.

Lu: It shifts the tuning effort from figuring out where to draw the line to just optimizing how you set up that initial token competition, which is much cleaner.

Xiaomi AI Lab · Nanyang Technological University

cs.CL, cs.AI

Submitted: 2026-10-08

Updated: 2026-10-08

Code: https://github.com/OkinoYu/CoTSReasoning

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 91/100

The gist: The gist The Competitive Token Selection (CoTS) framework reframes latent reasoning from a binary mode selection into continuous token-level allocation using top-two probability competition to

Key concepts

Latent Reasoning: This involves using continuous soft embeddings or hidden states instead of just picking a single discrete token at each step. The goal is to capture richer intermediate representations and allow the model to explore multiple plausible reasoning paths simultaneously, rather than committing to one path immediately.
Competitive Token Selection (CoTS): A training-free framework that replaces hard switching with continuous allocation. It compares the probability of the two most likely next tokens; a dynamic weight is then assigned between the sampled token and its strongest alternative based on this competition, controlling how much soft exploration to use.
Discrete Anchor and Phase Lock: This mechanism ensures stability during reasoning. The mixed input vector is bounded by a convex combination involving the sampled discrete token, preventing representation drift. A phase lock further controls when the model transitions from complex reasoning back to emitting final, discrete answers.

Terminology

Latent Reasoning — 1× in this paper · explained in 3 episodes

Summary

The gist The Competitive Token Selection (CoTS) framework reframes latent reasoning from a binary mode selection into continuous token-level allocation using top-two probability competition to dynamically balance discrete commitment and soft exploration.

Method Overview

Standard generation samples a single discrete token yt ∼ pt and feeds its embedding E(yt) ∈ R d into the next step, which discards alternative reasoning possibilities
Training-free soft reasoning methods collect the top-k candidate tokens St = Top-k candidate tokens form a soft representation while top-two competition controls its interpolation with the sampledtoken embedding, preserving at least half of the weight on the discrete anchor
COTS blends the sampled discrete token embedding with the soft candidate mixture: xt+1 = (1 − αet)E(yt) + αetst, where αet ∈ [0, 0.5] is a dynamic mixing weight that controls the input vector xt+1 for the next step
Let vt,1 and vt,2 denote the two most probable candidates, with pt(vt,1) ≥ pt(vt,2)

The competition is quantified by the ratio ct = pt(vt,2)/pt(vt,1), and the mixing weight αt is defined as αt = ct / (1 + ct)

This allocation interpretation states that when preferred candidate dominates, αt approaches zero and input remains close to the sampled discrete token

Discrete Anchor and Phase Lock

The mixed input is written as a convex combination over St ∪ Yt, where wt(yt) = (1 − αet) + αet1[yt ∈ St]qt(yt) ≥ 1 − αet ≥ 0.5
This bounding prevents representation drift because∥xt+1 − E(yt)∥ = αet∥st − E(yt) ≤ 0.5∥st − E(yt)
A Phase Lock for Final Answers is introduced using binary control flags αet = rt(1 − dt)αt, where rt ∈ 0, 1 indicates whether the model is in the reasoning stage (switching from 1 to 0 immediately once is emitted), and dt ∈ 0, 1 protects designated math operators or formatting tokens

Results and Conclusion

COTS consistently outperforms standard CoT and state-of-the-art training-free latent reasoning methods across six reasoning benchmarks, achieving superior performance on Qwen3.5-9B and Gemma-4-12B-IT The framework suggests that the boundary between commitment and exploration need not be fixed but can evolve continuously with the model’s own uncertainty

References

[1] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou

[2] Zhen Zhang, Xuehai He, Weixiang Yan, Ao Shen, Chenyang Zhao, and Xin Eric Wang

[3] Yufan Zhuang, Liyuan Liu, Chandan Singh, Jingbo Shang, and Jianfeng Gao

[4] Dachuan Shi, Abedelkadir Asi, Keying Li, Xiangchi Yuan, Leyan Pan, Wenke Lee, and Wen Xiao

[5] Renyu Fu and Guibo Luo

[6] Xuan Li, Yining Wang, Yuchen Liu, Guanjun Liu, Delai Qiu, Shengping Liu, Jiaen Liang, Wei Huang, Jun Yu, and Junnan Zhu

[7] Gemma Team

[8] Qwen Team

[9] Shibo Hao, Sainbayar Sukhbaatar, DiJia Su, Xian Li, Zhiting Hu, Jason Weston, and Yuandong Tian

[10] Junhong Wu, Jinliang Lu, Zixuan Ren, Gangqiang Hu, Zhi Wu, Dai Dai, and Hua Wu

[11] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman

[12] Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe

[13] David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R.

Improvements for AI systems

The improved system can dynamically balance discrete commitment and soft exploration at each reasoning step by using top-two probability competition to dynamically balance discrete commitment and soft exploration. This allows for a continuous allocation of latent information rather than a binary mode selection, as described in the abstract.
The system introduces a mechanism that ensures stability by guaranteeing a minimum level of discrete anchoring: the coefficient assigned to the sampled token is wt(yt) = (1 − αet) + αet1[yt ∈ St]qt(yt) ≥ 1 − αet ≥ 0.5. This prevents representation drift by ensuring at least half of the total interpolation weight remains explicitly anchored to the sampled discrete token.
The system can adapt its reliance on latent reasoning based on local uncertainty: "When one candidate clearly dominates, CoTS keeps the representation close to the sampled discrete token to preserve a stable reasoning trajectory; as the competition strengthens, it increases the contribution of the soft representation." This provides a continuous signal for how much latent information should influence each step.
The system achieves superior performance across diverse model families and scales: COTS consistently improves overall reasoning performance over standard CoT and competitive training-free latent reasoning baselines, achieving high accuracy on models like Gemma-4-12BIT (86.76%). This demonstrates the framework's transfer across distinct model families and architectures.