Skip to content

Small models and on-device inference

  • Inference
  • Specialist topic or differentiator
  • About 2 h to learn
  • 3 vetted resources
Build a plan that includes this

What this topic covers

Not every feature needs a frontier model: small models of a few billion parameters or less run on phones and laptops, cost almost nothing per call and keep data local. Interviewers ask when a small model is good enough, how you’d design an on-device assistant, and how you’d split work between device and cloud. A strong answer covers memory and latency budgets, aggressive quantization (around 2–4 bits per weight) and what it costs in quality, task-specific LoRA adapters on a shared base, why data quality matters more at small scale, routing hard requests to a server model, and evaluating on the target hardware rather than a GPU box.

Builds on

You’re ready when you can

What interviewers check, from the resources below.

  • Estimate the memory for a 3B model at 16, 4 and about 3.7 bits per weight
  • Explain how per-task adapters share one base model on device
  • Explain why the system pairs an on-device model with a server model
  • Say which latency numbers you’d measure on the target device
  • Explain why data quality matters more for small models
  • Read a memory-footprint chart and pick a model size for a phone
  • Name the formats used to run small models locally (ONNX, GGUF, WebGPU)
  • Argue for and against small models in an agent with three repetitive tools
  • Describe how you’d use logged agent calls to fine-tune a small replacement
  • Explain a mixed design where a large model handles only the hard turns

What to study, in order

Each link opened and checked by hand; the date says when.

  1. Start here

    Introducing Apple’s On-Device and Server Foundation Models (opens in a new tab)

    Apple Machine Learning Research · Apple · 35 min · Free · checked 22 Oct 2026

    Why
    A real shipped design for an on-device assistant model: about 3B parameters, under 4 bits per weight, swappable LoRA adapters and a server model for harder work. It is a ready-made outline for “design an on-device assistant”.
    What to read
    Read “Optimization” (palettization, bits per weight, latency on device) and “Model Adaptation” (adapters per task) closely, then skim “Pre-Training”, “Post-Training” and the summarization part of “Performance and Evaluation”.
    What to skip
    Skip the safety and harmfulness evaluation charts and the benchmark tables.

  2. Free alternative

    SmolLM - blazingly fast and remarkably powerful (opens in a new tab)

    Hugging Face · Loubna Ben Allal et al. · 30 min · Free · checked 22 Oct 2026

    Why
    Shows how 135M to 1.7B models get surprisingly good, mostly through data quality, and how they run locally on phones and in the browser.
    What to read
    Read “Introduction”, “Data curation” (Cosmopedia v2, FineWeb-Edu, Stack-Edu-Python), “Evaluation” and “How to run locally ?” with its memory footprint numbers.
    What to skip
    Skip “Training: Hyperparameters choice” and “Experiments”.

  3. Go deeper

    Small Language Models are the Future of Agentic AI (opens in a new tab)

    arXiv · Peter Belcak et al. · 40 min · Free · checked 22 Oct 2026

    Why
    A widely discussed position paper from NVIDIA arguing that most agent calls are narrow and repetitive enough for small models. It gives you the argument for “when is a small model good enough?”
    What to read
    Read §2 Position, §3 Position Arguments (especially 3.1 sufficiently powerful, 3.2 more economical, 3.4 narrow functionality, 3.6 naturally heterogeneous), §5 Barriers to Adoption and §6 LLM-to-SLM Agent Conversion Algorithm.
    What to skip
    Skim §4 Alternative Views; skip §7 Call for Discussion and the appendix case studies unless curious.

Companies that test Inference most

Put Small models and on-device inference in your plan

Tell us your interview date and hours a day; it lands on the right day, in the right order.

Build my plan