PPO, the RLHF workhorse: the probability ratio, the clipped surrogate objective, value and entropy terms, GAE advantages, and why PPO-RLHF holds four models.