Multi-Token Prediction Done Right

Their proposed fix, Soft Multi-Token Prediction (Soft-MTP), uses a "KL gradient-recovery trick" to isolate and backpropagate only the useful self-distillation signal by reusing existing cross-entropy computations, requiring a one-line code change, no new parameters, and no measurable overhead. Across models from 167M to 1.2B parameters, Soft-MTP beat standard MTP on next-token perplexity, speculative decoding speed, and zero-shot accuracy at every scale, and improved math and code reasoning after fine-tuning at 1.2B. It also matched or exceeded direct KL divergence minimization on most metrics and supports both DeepSeek- and Meta-style MTP. The result matters because it offers what the authors call a "free lunch" for MTP optimization, potentially making faster inference and stronger downstream performance available to any team already using MTP pre-training. Watch for adoption of the released code and whether the gains hold at frontier model scales.