Skip to content

Pretraining objectives and scaling laws

  • Transformers
  • Specialist topic or differentiator
  • About 1 h 50 min to learn
  • 6 vetted resources
Build a plan that includes this

What this topic covers

Pretraining a decoder-only LLM means minimising next-token cross-entropy over a huge text mixture, with the loss curve, learning-rate schedule (warmup, cosine or warmup-stable-decay) and data mix as the main levers. Scaling laws describe how that loss falls predictably with parameters, tokens and compute; the Chinchilla result shifted the field from very large under-trained models to roughly equal scaling of parameters and data, and labs now deliberately over-train small models because inference cost dominates. Applied-LLM interviews rarely ask you to pretrain anything, but they do ask what the objective is, how to read a loss curve, why a 3B model trained on 11T tokens can beat a bigger one, and how you would run cheap ablations before committing to a big run.

Builds on

You’re ready when you can

What interviewers check, from the resources below.

  • Explain when pretraining your own model is justified versus fine-tuning an open one
  • Describe how to set up a small ablation run (model size, token budget, data mix) and what makes its result trustworthy
  • Compare cosine and warmup-stable-decay schedules and say why WSD makes ablations cheaper
  • List the main data-mixture levers (web, code, math) and how staged mixing changes across training
  • State the objective being optimised and the rough batch size, sequence length and learning rate used for a 3B model
  • Explain why the data mix shifts toward code and math in later stages
  • Describe what a WSD (warmup-stable-decay) schedule looks like and when the decay phase starts
  • Compute tokens-per-parameter for this run and compare it with the Chinchilla rule of thumb
  • State the paper's headline: parameters and training tokens should scale roughly equally for a fixed compute budget
  • Give the approximate tokens-per-parameter rule of thumb and the Gopher versus Chinchilla comparison that motivated it

What to study, in order

Each link opened and checked by hand; the date says when.

  1. Start here

    The Smol Training Playbook: The Secrets to Building World-Class LLMs (opens in a new tab)

    Hugging Face · Loubna Ben Allal, Lewis Tunstall, Nouamane Tazi, Elie Bakouch et al. · 50 min · Free · checked 21 Oct 2026

    Why
    Interviewers want to hear how a real lab decides whether to pretrain at all, picks a baseline, and runs cheap ablations before a big run; this is the current reference for that process.
    What to read
    Read “Training Compass: Why → What → How” (all subsections, especially “Ablation Setup” and “Rules of Engagement”) and “Every Big Model Starts with a Small Ablation”, then skim “Architecture Choices” for the learning-rate schedule comparison.
    What to skip
    The later infrastructure and post-training chapters; and the PDF download, which is Pro-only.

  2. Free alternative

    SmolLM3: smol, multilingual, long-context reasoner (opens in a new tab)

    Hugging Face Blog · Elie Bakouch, Loubna Ben Allal, Leandro von Werra, Thomas Wolf et al. · 20 min · Free · checked 21 Oct 2026

    Why
    A concrete, recent pretraining recipe with real numbers (batch size, learning rate, schedule, three-stage data mix) is the fastest way to answer “walk me through how an LLM is pretrained”.
    What to read
    Read “Pretraining” → “Architecture and training details” and “Data mixture and training stages” only.
    What to skip
    “Mid-training”, “Post-training”, “Evaluation” and “How to run locally”; those belong to fine-tuning and long-context topics.

  3. Go deeper

    Training Compute-Optimal Large Language Models (opens in a new tab)

    arXiv · Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch et al. (DeepMind) · 45 min · Free · checked 21 Oct 2026

    Why
    “What is Chinchilla-optimal and why do labs now over-train?” is the standard scaling-laws question, and knowing the three estimation approaches shows you read the paper rather than a tweet.
    What to read
    Read “1 Introduction”, then “3 Estimating the optimal parameter/training tokens allocation” (3.1–3.4), then “5 Discussion & Conclusion”.
    What to skip
    “4 Chinchilla” benchmark subsections (4.2.1–4.2.7) and all appendices.

  4. Go deeper

    Scaling Laws for Neural Language Models (opens in a new tab)

    arXiv · Jared Kaplan et al. · 45 min · Free · checked 22 Oct 2026

    Why
    When an interviewer asks how loss scales with parameters, data and compute, this is the original source, and the follow-up is usually how Chinchilla changed its compute-optimal answer.
    What to read
    Read §1.1 and §1.2 (the summary of findings and the power-law equations) with Figure 1. Then §3 on why shape matters less than size, and §6.1 on how to split a compute budget, so you can contrast it with Chinchilla.
    What to skip
    Skip §5 (critical batch size and early stopping), §7 related work and Appendices B and D unless you want the fitting details.

  5. Go deeper

    The Llama 3 Herd of Models (opens in a new tab)

    arXiv · Aaron Grattafiori et al. · 90 min · Free · checked 22 Oct 2026

    Why
    This is the most detailed public account of how a frontier open model was pretrained and post-trained, and interviewers use it for questions on data mixes, scaling-law forecasts and modern post-training.
    What to read
    Read §3.1 (3.1.1 web data curation, 3.1.2 data mix, 3.1.3 annealing), §3.2.1 on using scaling laws to choose model size, and §3.4 on the training recipe and long-context stage. Then read §4.1 (reward model, SFT, DPO, model averaging, iterative rounds) and §4.2.3 on data quality control.
    What to skip
    Skip §3.3 infrastructure unless you’re preparing for distributed training, §4.3 per-capability sections, §5 benchmark tables, and §7–8 on vision and speech.

  6. Go deeper

    nanochat (opens in a new tab)

    GitHub · karpathy · 75 min · Free · checked 22 Oct 2026

    Why
    nanochat puts every stage of building a chat model in one small codebase: tokenizer, pretraining, SFT, RL, evaluation and a KV-cached inference engine. It’s the clearest way to answer “walk me through how an LLM is made, end to end”.
    What to read
    Read the README’s “Research” section on the single --depth dial and compute-optimal sizing, then runs/speedrun.sh top to bottom to see the stage order. Skim scripts/base_train.py (how depth sets width, batch and horizon), nanochat/loss_eval.py (bits per byte) and nanochat/engine.py (KV cache). Run runs/runcpu.sh on a laptop to watch the whole pipeline at toy scale.
    What to skip
    Skip the leaderboard table, the precision/dtype section, fp8.py and optim.py’s Muon details unless you’re interviewing for pretraining. You don’t need to rent the 8×H100 node.

Companies that test Transformers most

Put Pretraining objectives and scaling laws in your plan

Tell us your interview date and hours a day; it lands on the right day, in the right order.

Build my plan