Fine-tuning LLMs via policy gradient algorithms

Starting from first principles, I will derive (a variant of) the GRPO algorithm, one of the most widely used algorithms for post-training large language models. Then I will sketch how this algorithm is implemented at scale. Finally, I will briefly describe an important open problem known as training--inference mismatch.