ManySpace
Türkçe Sign in Sign up

AI & MLRecommendedAIML-LM-020

Scaling Laws and Compute-Optimal Training

Loss as a power law in parameters, data, and compute; how to split a compute budget between model size and tokens.

Difficulty
3 / 5
Importance
4 / 5
Study time
1–2 h
Kind of work
Mixed

Why it matters

The learner can use a scaling-law fit to choose a compute-optimal model size and token count for a given budget.

The route to this topic

Route AIML-LM-020
What Is a Model? Tasks: Regression, Classification, and Beyond Features and Targets Classification Framing Linear Regression I: Prediction and Residuals Tokens and Tokenization Scores and Logits Loss and Fitting: Squared Error Multiclass Scores and Softmax The Artificial Neuron Autoregression Objective Functions and Empirical Risk Activation Functions Next-Token Prediction Gradient Descent in Machine Learning Likelihood The Multilayer Perceptron Cross-Entropy Loss Stochastic and Mini-batch Optimization Forward Propagation The Language-Modeling Objective Training Dynamics Pretraining Scaling Intuition Scaling Laws and Compute-Optimal Training