Tokenization Explorer
Type a sentence and watch a simplified toy tokenizer split it into pieces: suffixes peel off, numbers become digits, and tokens turn out to be neither words nor characters.
Simplified BPE-style model, fully local, nothing leaves your browser
Try this
- Scrub the merge steps of “okullar” and watch the suffix survive.
- Type a number with punctuation and see digits split apart.
- Find a word that shatters into single letters.
How tokenization works here
Real language models read text as integer token IDs, not characters. This lab runs a simplified byte-pair (BPE) model with about 25 hand-written merges so you can watch the process: text is lowercased, split into segments, and each word's letters merge step by step into pieces.
The vocabulary is tiny on purpose — a teaching model, not a production tokenizer. Its token IDs do not match GPT or any other real system. What transfers is the core idea: token boundaries follow learned merges, so suffixes, punctuation, spaces, and digits each behave differently.
Understand the models behind the pieces
ManySpace turns AI mechanics like this into guided lessons.