Token-Level Off-Policy Learning for Faithful Generation Under Distribution Shift
Token-Level Off-Policy Labeling (TOPL) rethinks post-training by treating generation as a token correctness prediction task. Instead of relying on next-token prediction, it trains a model to classify each generated token as 'good' or 'bad' based on factual accuracy against the source document. This method uses low-rank adaptation to guide the model toward generating factually correct tokens and improves its ability to generalize when faced with new data.