How LLMs Break Down Language
The complete guide to tokenization - the first step in how large language models process text
What is Tokenization?
Tokenization is the process of breaking down raw text into smaller units called tokens, which serve as the fundamental building blocks that LLMs use to understand and generate language.
Why Tokenization Matters
- Converts unstructured text into structured numerical data
- Enables models to handle variable-length inputs
- Creates a vocabulary of meaningful units
The Tokenization Process Step-by-Step
Text Normalization
Standardizing the input text by converting to lowercase, removing extra spaces, and handling Unicode characters.
Normalized: "llm's process text"
Pre-tokenization
Initial splitting into words, punctuation, and special tokens using rules or whitespace.
Pre-tokens: ["Don", "'", "t", "split", "me", "!"]
Token Algorithm
Applying the tokenization algorithm (BPE, WordPiece, etc.) to split into final tokens.
Tokens: ["un", "happiness"]
Try It Yourself
Tokens will appear here...
Tokenization Algorithms
Byte Pair Encoding (BPE)
Iteratively merges most frequent character pairs to create a vocabulary of subword units.
WordPiece
Similar to BPE but uses likelihood rather than frequency to determine merges.
- Used in BERT models
- Handles unknown words better
- Prioritizes meaningful subword units
| Algorithm | Used In | Strengths | Weaknesses |
|---|---|---|---|
| BPE | GPT models | Simple, efficient, handles rare words | May split words unnaturally |
| WordPiece | BERT, DistilBERT | Better for masked language modeling | More complex training |
| SentencePiece | T5, ALBERT | Language agnostic, no pre-tokenization | Larger vocabulary sizes |
Challenges in Tokenization
Out-of-Vocabulary Words
When encountering unknown words, models must rely on subword units which may not capture the word's meaning accurately.
Contextual Ambiguity
Same token can have different meanings based on context, which pure tokenization doesn't resolve.
Same token ID, different meanings
Sequence Length
Models have fixed maximum token limits, requiring truncation or chunking of long texts.
Longer documents must be split
Multilingual Support
Languages with different scripts or no word boundaries pose unique tokenization challenges.
No spaces between words
Mastering Tokenization
Tokenization is the critical first step that enables LLMs to process human language. Understanding it provides insight into how these models work and their limitations.
Key Takeaways
- Tokenization converts text to numerical representations
- Subword tokenization balances vocabulary size and coverage
- Different algorithms suit different model architectures