AI & MLRecommendedAIML-SYS-002
Latency and Throughput
Fast answers vs many answers: batching trades one for the other.
- Difficulty
- 3 / 5
- Importance
- 4 / 5
- Study time
- 1–2 h
- Kind of work
- Mixed
Why it matters
The learner can trade latency against throughput via batch size on one serving curve.