Skip to content

Embeddings basics: word2vec, contrastive learning, what embeddings capture

  • ML fundamentals
  • Asked in most interview loops
  • About 70 min to learn
  • 3 vetted resources
Build a plan that includes this

What this topic covers

An embedding is a dense vector learned so that similar things end up close together. This topic covers where those vectors come from: word2vec-style prediction tasks with negative sampling, embedding layers trained inside a larger network, and contrastive objectives (InfoNCE and friends) that pull positive pairs together and push negatives apart, which is how modern sentence and retrieval embedders are trained. Interviewers ask what an embedding actually encodes, why static word vectors fail on ambiguous words, why cosine distance is the usual metric, how negatives and batch size affect contrastive training, and how you would pick or fine-tune an embedding model for a RAG system. This is the conceptual floor under embeddings-vector-search and rag-retrieval.

Leads to

You’re ready when you can

What interviewers check, from the resources below.

  • Explain why a language-modelling task produces vectors where similar words are close
  • Describe the skipgram training example and why negative sampling turns it into a cheap classification problem
  • State what the two embedding matrices are and which one is kept after training
  • Explain how window size and the number of negatives change what the embeddings capture
  • Explain the two problems with one-hot inputs that embeddings solve
  • Compare an embedding learned inside a task network with a pretrained static embedding
  • Explain why a static word vector fails on a word with several meanings and how contextual embeddings fix it
  • Write down the InfoNCE loss and explain what the positive and the denominator each represent
  • Explain why large batches and hard negatives matter for contrastive training
  • Describe how a sentence embedder is trained from NLI pairs versus with unsupervised augmentation

What to study, in order

Each link opened and checked by hand; the date says when.

  1. Start here

    The Illustrated Word2vec (opens in a new tab)

    jalammar.github.io · Jay Alammar · 40 min · Free · checked 21 Oct 2026

    Why
    “How are embeddings trained?” is the standard warm-up before any RAG or retrieval question, and word2vec with negative sampling is the canonical answer interviewers expect you to sketch.
    What to read
    Read “Word Embeddings” through “Window Size and Number of Negative Samples”; the sections “Skipgram”, “Negative Sampling” and “Skipgram with Negative Sampling (SGNS)” are the core.
    What to skip
    “Personality Embeddings: What are you like?” if you already have the vector intuition, and “References & Further Readings”.

  2. Free alternative

    Embeddings (Machine Learning Crash Course) (opens in a new tab)

    Google for Developers · Google · 30 min · Free · checked 21 Oct 2026

    Why
    A short, vendor-neutral grounding in what an embedding space is and the three ways to obtain one, which is the vocabulary interviewers use when they ask you to compare static and contextual embeddings.
    What to read
    Read the “Introduction” page (“Pitfalls of sparse data representations”), then the sub-pages “Embedding space and static embeddings” (“Real-world embedding spaces”, “Static embeddings”) and “Obtaining embeddings” (“Dimensionality reduction techniques”, “Training an embedding as part of a neural network”, “Contextual embeddings”).
    What to skip
    “Interactive exercises” and “Test your knowledge” unless you want a quick check; “What's next”.

  3. Go deeper

    Contrastive Representation Learning (opens in a new tab)

    Lil'Log · Lilian Weng · 45 min · Free · checked 21 Oct 2026

    Why
    Every modern sentence and retrieval embedder (and CLIP) is trained with a contrastive loss, so senior-level follow-ups go straight to InfoNCE, negatives and batch size.
    What to read
    Read “Contrastive Training Objectives” (especially “Contrastive Loss”, “Triplet Loss”, “InfoNCE” and “Common Setup”), “Key Ingredients”, and “Language: Sentence Embedding”.
    What to skip
    Most of “Vision: Image Embedding” (SimCLR, MoCo, SwAV etc.); read only the “CLIP” subsection if you work with multimodal retrieval. Skip “References”.

Companies that test ML fundamentals most

Put Embeddings basics in your plan

Tell us your interview date and hours a day; it lands on the right day, in the right order.

Build my plan