Skip to content

Distillation and synthetic data

  • Transformers
  • Specialist topic or differentiator
  • About 1 h 30 min to learn
  • 4 vetted resources
Build a plan that includes this

What this topic covers

Distillation trains a smaller student model to copy a larger teacher, either by matching the teacher’s output probabilities (logit-level) or by fine-tuning on text the teacher generated (sequence-level), the way DeepSeek distilled R1 into small Qwen and Llama models. Synthetic data is the broader practice of generating training examples with a model and then filtering them hard. Interviewers raise it as a cost question: “this works on the big model, how do we make it cheap?” A strong answer covers the distillation variants, how you generate, filter and deduplicate synthetic data, failure modes such as copying the teacher’s style without its accuracy, and licence limits on using a provider’s outputs.

Builds on

You’re ready when you can

What interviewers check, from the resources below.

  • Distinguishes distillation from a stronger model from self-improvement
  • Describes a generate, filter, deduplicate pipeline with concrete filters
  • Names limits: self-improvement can amplify errors, and teacher licence terms apply
  • Explains why Orca-style explanation traces beat bare answers
  • Contrasts off-policy (teacher samples) with on-policy (student samples, teacher scores each token)
  • Explains why imitating teacher samples can copy style without accuracy
  • Explains why reverse KL is used and what dense per-token feedback buys
  • Frames distillation versus RL as a compute and cost trade-off
  • Writes the temperature-scaled softmax and the combined soft-plus-hard loss
  • Explains why soft targets carry more information than one-hot labels

What to study, in order

Each link opened and checked by hand; the date says when.

  1. Start here

    How to Generate and Use Synthetic Data for Finetuning (opens in a new tab)

    eugeneyan.com · Eugene Yan · 45 min · Free · checked 21 Oct 2026

    Why
    It maps the field into distillation from a stronger model and self-improvement, which gives you a clear structure when asked how you’d build a training set without hand labelling.
    What to read
    Read “Self-Improvement vs. Distillation”, then “Distillation techniques” focusing on Alpaca, WizardLM (Evol-Instruct), Orca and Phi-1. Then read “Synthetic data for instruction-tuning” (Instruction Backtranslation, SPIN, ReST^EM). Note each filtering step as you go: deduplication, model grading, quality classifiers.
    What to skip
    Skim “Synthetic data for preference-tuning” and “Synthetic data for pretraining” (AlphaGeometry, WRAP) unless the role involves pretraining.

  2. Free alternative

    On-Policy Distillation (opens in a new tab)

    Thinking Machines Lab · Kevin Lu · 40 min · Free · checked 21 Oct 2026

    Why
    It compares sequence-level distillation (fine-tuning on teacher samples), RL and per-token teacher supervision on the same reasoning task with compute numbers, which is exactly the cost reasoning interviewers want.
    What to read
    Read “On-policy distillation — best of both worlds” including “Loss function: reverse KL” and the pseudocode, then “Distillation for reasoning” (off-policy distillation, RL, on-policy distillation) and the first two parts of “Discussion”.
    What to skip
    Skip “Distillation for personalization” unless you care about continual learning, and the last two “Discussion” subsections.

  3. Go deeper

    Distilling the Knowledge in a Neural Network (opens in a new tab)

    arXiv · Geoffrey Hinton, Oriol Vinyals, Jeff Dean · 30 min · Free · checked 21 Oct 2026

    Why
    This is the original paper on matching soft targets, and you’ll be asked what temperature does and why soft labels carry more information than hard ones.
    What to read
    Read the introduction and the section on distillation (soft targets, temperature and the T² gradient scaling), then the MNIST and speech results.
    What to skip
    Skip the section on ensembles of specialist models; it rarely comes up for LLM roles.

  4. Go deeper

    Self-Instruct: Aligning Language Models with Self-Generated Instructions (opens in a new tab)

    arXiv · Yizhong Wang et al. · 30 min · Free · checked 22 Oct 2026

    Why
    Self-Instruct is the template for bootstrapping instruction data from a model, and interviewers expect you to know its pipeline and the filtering that keeps the data diverse.
    What to read
    Read §2.2 with Figure 2 (seed tasks, instruction generation, classification check, input-first versus output-first instance generation, ROUGE-L filtering). Then read §3.2–3.3 on diversity and quality and §4.5 on data size and quality.
    What to skip
    Skip §4.3–4.4 benchmark details beyond the headline, §5 related work and Appendix B.

Companies that test Transformers most

Put Distillation and synthetic data in your plan

Tell us your interview date and hours a day; it lands on the right day, in the right order.

Build my plan