AI & MLRecommendedAIML-LM-021
Pretraining Data: Sourcing, Filtering, and Mixtures
Where pretraining text comes from and how deduplication, quality filtering, and domain mixture shape the model.
- Difficulty
- 3 / 5
- Importance
- 4 / 5
- Study time
- 1–2 h
- Kind of work
- Mixed
Why it matters
The learner can outline a pretraining data pipeline and name the risk each stage (dedup, quality filter, toxicity filter, mixture weights) addresses.