AI & MLRecommendedAIML-SYS-016
Distributed Training and Parallelism
Data, tensor, pipeline, and sharded parallelism, and the communication each requires.
- Difficulty
- 4 / 5
- Importance
- 4 / 5
- Study time
- 1–2 h
- Kind of work
- Mixed
Why it matters
The learner can choose a parallelism strategy for a model that does not fit on one device and explain its communication cost.