kaal:claim:7456999-015

The canonical RLHF pipeline consists of supervised fine-tuning, reward-model training on human preference comparisons, and policy optimization.

The canonical pipeline, established by Christiano et al. (2017) and extended by Ouyang et al. (2022), proceeds in three stages: supervised fine-tuning on demonstration data, reward model training on human preference comparisons under the Bradley-Terry likelihood, and policy optimization via Proximal Policy Optimization (Schulman et al. 2017).

Wulf A. Kaal, Reinforcement Learning from Reputation Feedback (2026), I. Introduction, p. 5
https://ssrn.com/abstract=7456999 · source PDF

Wulf A. Kaal, Reinforcement Learning from Reputation Feedback (2026). SSRN: https://ssrn.com/abstract=7456999

• the canonical RLHF pipeline described by the cited literature

definitionalsupport: arguedai-and-agentsresearch-methods

The quote above is an exact passage from the source PDF, whose sha256 is 01016fbbcc25cad205cebd59ea686d8dfee8ae9b632095d86ba6c9a900456a4b.
Verify the binding yourself: curl -s https://wulfkaal.github.io/claims/7456999-015.md | sha256sum

• Agreement: Ouyang et al.'s InstructGPT method uses supervised fine-tuning, reward-model training on labeler-ranked outputs, and PPO optimization against that reward model.