Qualification: In Gao, Schulman, and Hilton's synthetic proxy-versus-gold reward-model setup, optimizing an imperfect proxy too much can hinder ground-truth performance.
Gao, Schulman, and Hilton show the central Goodhart problem in reward-model optimization: an imperfect proxy can improve under optimization while ground-truth performance worsens. This supports the protected overoptimization claim in their controlled synthetic setup. The result should not be overstated. Their gold-standard reward model stands in for humans, and the abstract does not establish that degradation is monotonic in realized KL distance from the reference policy.
Affirmed commentary position. This record extends a source-bound scholarly claim but is not a verbatim paper claim.
Holds when
Current debate
Scholarly basis
Evidence and mapping
Topics
ai-and-agentsrisk-and-incentivesempirical-evidence
Provenance
Verify