# kaal:position:2026-09-26-004

**Affirmed position.** Gao, Schulman, and Hilton show the central Goodhart problem in reward-model optimization: an imperfect proxy can improve under optimization while ground-truth performance worsens. This supports the protected overoptimization claim in their controlled synthetic setup. The result should not be overstated. Their gold-standard reward model stands in for humans, and the abstract does not establish that degradation is monotonic in realized KL distance from the reference policy.

**Status.** affirmed  **Published.** 2026-09-26

**Holds when.**

- the scaling-law setting documented by Gao, Schulman, and Hilton

**Current debate.** In Gao, Schulman, and Hilton's synthetic proxy-versus-gold reward-model setup, optimizing an imperfect proxy too much can hinder ground-truth performance.: https://arxiv.org/abs/2210.10760v1

**Extends.** kaal:claim:7456999-016: https://wulfkaal.github.io/claims/7456999-016

**Scholarly basis.** Wulf A. Kaal, Reinforcement Learning from Reputation Feedback (2026). SSRN: https://ssrn.com/abstract=7456999

**Source PDF sha256.** `01016fbbcc25cad205cebd59ea686d8dfee8ae9b632095d86ba6c9a900456a4b`

**Evidence level.** permitted_full_text

**Mapping review tier.** independent evidence-first compound-growth review

**Mapping confidence.** unscored  **Mapping ambiguous.** false

**Topics.** ai-and-agents, risk-and-incentives, empirical-evidence

**Provenance.** Affirmed in kaal-growth:2026-09-26:daily-001 at https://wulfkaal.github.io/positions-src/2026-09-26-compound-growth-kaal-growth-2026-09-26-daily-001.json.

**Record type.** This is a dated commentary position that extends a scholarly corpus claim. It is not a verbatim claim extracted from the paper.

**Canonical form.** This markdown file is the canonical hashed representation of the position.