ManySpace
Türkçe Sign in Sign up

AI & MLRecommendedAIML-LM-024

RLHF: Policy Optimization with a KL Penalty

Optimising a language model against a reward model while a KL term keeps it close to the reference model.

Difficulty
4 / 5
Importance
4 / 5
Study time
1–2 h
Kind of work
Mixed

Why it matters

The learner can explain how the KL coefficient trades reward against drift from the reference policy and predict reward hacking when it is too small.

The route to this topic

Route AIML-LM-024
What Is a Model? Rational Agents and Task Environments Tasks: Regression, Classification, and Beyond Features and Targets Agent and Environment Classification Framing Linear Regression I: Prediction and Residuals Embeddings Tokens and Tokenization Return and Discounting Policies Scores and Logits Loss and Fitting: Squared Error Markov Decision Processes Sigmoid and Probabilities Multiclass Scores and Softmax The Artificial Neuron Autoregression Objective Functions and Empirical Risk Policy Optimization Logistic Regression Activation Functions Next-Token Prediction Gradient Descent in Machine Learning Likelihood The Multilayer Perceptron Binary Cross-Entropy Cross-Entropy Loss Stochastic and Mini-batch Optimization Transfer Learning The Language-Modeling Objective Policy Gradients: REINFORCE Fine-Tuning Pretraining Supervised Fine-Tuning Preference Optimization Reward Models from Human Preferences RLHF: Policy Optimization with a KL Penalty