# Tokenization Explorer

Canonical: https://manyspace.aporia.systems/tools/tokenization
Updated: 2026-10-13

Type a sentence and watch a simplified toy tokenizer split it into pieces: suffixes peel off, numbers become digits, and tokens turn out to be neither words nor characters.

## Try this

- Scrub the merge steps of “okullar” and watch the suffix survive.
- Type a number with punctuation and see digits split apart.
- Find a word that shatters into single letters.

## How tokenization works here

Real language models read text as integer token IDs, not characters. This lab runs a simplified byte-pair (BPE) model with about 25 hand-written merges so you can watch the process: text is lowercased, split into segments, and each word's letters merge step by step into pieces.

The vocabulary is tiny on purpose — a teaching model, not a production tokenizer. Its token IDs do not match GPT or any other real system. What transfers is the core idea: token boundaries follow learned merges, so suffixes, punctuation, spaces, and digits each behave differently.

## Related tools

- [Gradient Descent Playground](https://manyspace.aporia.systems/tools/gradient-descent)
- [Loss Function Explorer](https://manyspace.aporia.systems/tools/loss-functions)
- [k-Means Visualizer](https://manyspace.aporia.systems/tools/k-means)
- [Logistic Regression](https://manyspace.aporia.systems/tools/logistic-regression)
- [Forward Pass](https://manyspace.aporia.systems/tools/neural-network)
- [Backpropagation](https://manyspace.aporia.systems/tools/backpropagation)
- [Attention Visualizer](https://manyspace.aporia.systems/tools/attention)