Generalization and Model Selection

Diagnosis

1. Train, Validation, and Test ErrorsThree numbers, three jobs: measuring fit, selecting, and honestly reporting.1–2 h
2. Overfitting and UnderfittingThe gap you can see: flexibility drives train and test errors apart.1–2 h
3. Bias-Variance Trade-offError, decomposed: systematic miss plus jitter, and why both cannot shrink together.1–2 h
4. Model CapacityWhat flexibility is made of: parameters, degrees of freedom, and function-class size.1–2 h

Control

1. Regularization: RidgePaying for size: the lambda penalty shrinks weights and calms the fit.1–2 h
2. Regularization: Lasso and SparsityL1 geometry selects: corners zero out weights where circles only shrink them.1–2 h
3. Early StoppingTime as regularization: halting the fit where validation bottoms out.1–2 h
4. Data AugmentationEncoding invariances by transforming training examples: more effective data without more labels.1–2 h
5. Ensembles: Bagging and AveragingAveraging many high-variance models reduces variance; bootstrap resampling makes the models different.1–2 h

Selection

1. Cross-ValidationHonest error estimates by rotation: k folds, k trials, one averaged verdict.1–2 h
2. Hyperparameters and SelectionKnobs outside the fit: parameters vs hyperparameters, grids, and validation discipline.1–2 h