kaal:claim:7456999-016

Reward-model overoptimization can raise proxy reward while degrading true reward as optimized policies move farther from the reference policy.

Gao, Schulman, and Hilton (2023) demonstrated that reward model overoptimization follows predictable scaling laws: as KL divergence between the optimized policy and the reference policy grows, the proxy reward continues to rise while the true reward degrades.

Wulf A. Kaal, Reinforcement Learning from Reputation Feedback (2026), I. Introduction, p. 5
https://ssrn.com/abstract=7456999 · source PDF

Wulf A. Kaal, Reinforcement Learning from Reputation Feedback (2026). SSRN: https://ssrn.com/abstract=7456999

• the scaling-law setting documented by Gao, Schulman, and Hilton

empiricalsupport: evidencedai-and-agentsrisk-and-incentivesempirical-evidence

The quote above is an exact passage from the source PDF, whose sha256 is 01016fbbcc25cad205cebd59ea686d8dfee8ae9b632095d86ba6c9a900456a4b.
Verify the binding yourself: curl -s https://wulfkaal.github.io/claims/7456999-016.md | sha256sum

• Qualification: In Gao, Schulman, and Hilton's synthetic proxy-versus-gold reward-model setup, optimizing an imperfect proxy too much can hinder ground-truth performance.