Next-token Preference: Scaling Unsupervised Reward Modeling
The method introduces Next-Token Preference (NTP), an unsupervised way to create reward models by learning from raw web text structure. NTP constructs preference pairs by using a prefix and its original continuation as the chosen response, contrasting it with nearby continuations as negatives. Training on 11M math tokens shows steady gains on RewardBench, proving that transferable preference signals exist in large web data, reducing reliance on costly human feedback.