MathematicsOptionalMTH-INFO-343
Entropy & Mutual Information
joint/conditional entropy, KL divergence, chain rules
- Difficulty
- 4 / 5
- Importance
- 4 / 5
- Study time
- 6.5–12 h
- Kind of work
- Theory
Why it matters
Quantifies information and limits of compression/communication; the theoretical foundation of modern data science and ML.
Where it is used
Lossless/lossy compression, error correction, ML loss functions (cross-entropy), cryptography, communications.
How you know you have learned it
Explain/derive key result from memory + solve 2 non-routine checks.
Resources
- Elements of Information Theory (Cover & Thomas) — Cover & Thomas Book