Home / Glossary / Tokenization
What is Tokenization?
Tokenization splits raw text into the sub-word units (tokens) that LLMs actually process. Different models use different tokenisers; the same text counts as a different number of tokens depending on the model.
Detail
Most modern LLMs use byte-pair encoding (BPE) or similar — frequent character sequences become single tokens, rare ones become multi-token. English averages ~4 characters per token; CJK languages average closer to 1.
Tokenisation affects pricing (per-token billing) and context-window usage. Code is usually tokenised more efficiently than prose. Special characters and emojis can take many tokens.
Related terms