Skip to content

Tokenization and BPE

  • Transformers
  • Asked often
  • About 1 h 30 min to learn
  • 3 vetted resources
Build a plan that includes this

What this topic covers

Models never see text, only integer IDs, and the tokenizer decides how text is chopped into those IDs. Byte pair encoding starts from raw UTF-8 bytes and repeatedly merges the most frequent adjacent pair into a new token until it reaches a target vocabulary size. Interviewers ask you to walk through training and encoding by hand, explain the vocabulary-size trade-off (sequence length vs embedding table size), and say why byte-level BPE never hits an unknown token. They also like to connect tokenization to model quirks: bad letter counting, weak arithmetic on numbers split oddly, and higher cost for non-English text.

You’re ready when you can

What interviewers check, from the resources below.

  • Explain why BPE starts from UTF-8 bytes rather than characters or words
  • Run three merge steps by hand on a short sentence and decode the result back
  • Describe what the trained tokenizer stores (vocabulary and ordered merges) and why merge order matters at encode time
  • Explain the trade-off a larger vocabulary makes between sequence length and embedding size
  • Explain how merge rules are learned from word frequencies
  • Tokenize an unseen word by applying learned merges in order
  • Say where normalization and pre-tokenization fit before BPE runs
  • Implement train, encode and decode for byte-level BPE and pass a round-trip test
  • Explain why GPT-style tokenizers split text with a regex before merging
  • Reproduce tiktoken's GPT-4 token IDs for a sample string

What to study, in order

Each link opened and checked by hand; the date says when.

  1. Start here

    Implementing A Byte Pair Encoding (BPE) Tokenizer From Scratch (opens in a new tab)

    sebastianraschka.com · Sebastian Raschka · 45 min · Free · checked 21 Oct 2026

    Why
    “How does a tokenizer work?” is a common warm-up, and walking through BPE merges on a tiny string shows you know what the model actually sees.
    What to read
    Read “1. The main idea behind byte pair encoding (BPE)” through “3.1 Training, encoding, and decoding”, tracing the worked example in “1.4 BPE algorithm example” by hand.
    What to skip
    “3.2 Saving and loading the tokenizer” and the book/newsletter promos; skim “3.3 Loading the original GPT-2 BPE tokenizer from OpenAI” only if you want the tiktoken comparison.

  2. Free alternative

    Byte-Pair Encoding tokenization (opens in a new tab)

    Hugging Face LLM Course · Hugging Face · 35 min · Free · checked 21 Oct 2026

    Why
    A shorter route to the same interview answer: how BPE learns merges and how it applies them to new words.
    What to read
    Read “Training algorithm” and “Tokenization algorithm”, then run the code in “Implementing BPE”.
    What to skip
    Nothing major; the page is short. The video embeds repeat the text.

  3. Go deeper

    minbpe (opens in a new tab)

    GitHub · Andrej Karpathy · 150 min · Free · checked 21 Oct 2026

    Why
    Building a tokenizer that matches GPT-4's output byte for byte is the strongest proof you understand regex pre-splitting and merge ranks, which come up in inference and tokenizer-bug questions.
    What to read
    Read the README's “quick start” and “training”, then do Steps 1–3 of exercise.md (BasicTokenizer, RegexTokenizer, matching tiktoken's GPT-4 encoding). Peek at minbpe/basic.py and regex.py only when stuck.
    What to skip
    exercise.md Step 5 (sentencepiece / Llama 2) unless you're targeting a pretraining role; “community extensions” and “todos”.

Companies that test Transformers most

Put Tokenization and BPE in your plan

Tell us your interview date and hours a day; it lands on the right day, in the right order.

Build my plan