# Entropy & Mutual Information

Canonical: https://manyspace.aporia.systems/topics/entropy-mutual-information-mth-info-343
Updated: 2026-10-13

Mathematics · Information Theory · MTH-INFO-343

- Difficulty: 4 / 5
- Importance: 4 / 5
- Study time: 6.5–12 h

joint/conditional entropy, KL divergence, chain rules

## Why it matters

Quantifies information and limits of compression/communication; the theoretical foundation of modern data science and ML.

## Where it is used

Lossless/lossy compression, error correction, ML loss functions (cross-entropy), cryptography, communications.

## How you know you have learned it

Explain/derive key result from memory + solve 2 non-routine checks.

## What you need first

- [Sample Spaces & Events](https://manyspace.aporia.systems/topics/sample-spaces-events-mth-prob-196) (required)
- [Random Variables & Distributions](https://manyspace.aporia.systems/topics/random-variables-distributions-mth-prob-200) (required)

## What it opens

- [Channel Capacity](https://manyspace.aporia.systems/topics/channel-capacity-mth-info-345)
- [Variational Inference and the ELBO](https://manyspace.aporia.systems/topics/variational-inference-and-the-elbo-aiml-pgm-010)
- [Cross-Entropy Loss](https://manyspace.aporia.systems/topics/cross-entropy-loss-aiml-cls-008)
- [Perplexity](https://manyspace.aporia.systems/topics/perplexity-aiml-lm-018)

## Resources

- Elements of Information Theory (Cover & Thomas) — Cover & Thomas