AI & MLRecommendedAIML-SYS-014
Compute, Memory, and Cost Accounting
Counting FLOPs, parameters, activations, and optimiser state to estimate training and inference cost.
- Difficulty
- 3 / 5
- Importance
- 4 / 5
- Study time
- 1–2 h
- Kind of work
- Mixed
Why it matters
The learner can estimate training FLOPs and memory for a transformer from parameter count and token count.