AI & MLRecommendedAIML-ATT-009
Efficient Attention
Attention's quadratic cost and the fixes: IO-aware kernels, sparse and sliding windows, and linear approximations.
- Difficulty
- 4 / 5
- Importance
- 4 / 5
- Study time
- 1–2 h
- Kind of work
- Mixed
Why it matters
The learner can compute how attention memory grows with context length and explain what IO-aware tiling changes and what it leaves unchanged.