Skip to content

Multimodal and vision-language models

  • Transformers
  • Specialist topic or differentiator
  • About 2 h to learn
  • 3 vetted resources
Build a plan that includes this

What this topic covers

Vision-language models let an LLM take images as input. Most build on two ideas: CLIP-style contrastive training, which puts images and text in one embedding space, and a LLaVA-style design, where a vision encoder’s patch features go through a small projector and are fed to the LLM as tokens. The main alternative is cross-attention, as in Flamingo or Llama 3’s vision models, where the LLM attends to image features without putting them into its context. Interviewers ask how images become tokens, what that costs in context and latency, and how high-resolution images are handled by tiling or native dynamic resolution. A strong answer also connects this to products: CLIP embeddings for image search and multimodal RAG over documents.

Builds on

Leads to

You’re ready when you can

What interviewers check, from the resources below.

  • Can draw vision encoder → projector → LLM and say which parts are frozen at each training stage
  • Can explain how an image becomes patch tokens and roughly what that costs in context
  • Can compare unified-decoder and cross-attention designs on cost and quality
  • Can describe one way to handle high-resolution images (tiling or native dynamic resolution)
  • Can explain CLIP’s contrastive loss over a batch of image-text pairs
  • Can explain how CLIP embeddings power text-to-image search and zero-shot classification
  • Can describe Flamingo’s cross-attention approach as the alternative to feeding image tokens into the context
  • Can describe LLaVA’s CLIP encoder, projection layer and LLM
  • Can explain what is frozen and what is trained in each of the two stages
  • Can explain how text-only GPT-4 was used to make image instruction data

What to study, in order

Each link opened and checked by hand; the date says when.

  1. Start here

    Understanding Multimodal LLMs (opens in a new tab)

    Ahead of AI · Sebastian Raschka · 60 min · Free · checked 21 Oct 2026

    Why
    The standard question is “how do you get an image into an LLM?” This post sets out the two main answers, a unified decoder with a projector and cross-attention, and shows how real models choose between them.
    What to read
    Read all of section 2: 2.1 Method A (2.1.1 Understanding Image encoders, 2.1.2 The role of the linear projection module, 2.1.3 Image vs text tokenization) and 2.2 Method B. Then read section 3 on training, and 4.1 Llama 3, 4.3 NVLM and 4.4 Qwen2-VL to see how models handle high-resolution images.
    What to skip
    Skip Use cases of multimodal LLMs and 4.5–4.10 (Pixtral 12B to Janus) on a first pass.

  2. Free alternative

    Multimodality and Large Multimodal Models (LMMs) (opens in a new tab)

    huyenchip.com · Chip Huyen · 50 min · Free · checked 21 Oct 2026

    Why
    CLIP’s contrastive objective underpins image search, zero-shot classification and the vision encoders inside most VLMs, and you may be asked to explain it. This is a clear, free walkthrough of CLIP and Flamingo.
    What to read
    Read Part 2. For CLIP: its high-level architecture, Natural language supervision, Contrastive learning (the classifier, language model and contrastive objectives compared) and CLIP applications, including text-based image retrieval. For Flamingo: the Perceiver Resampler and GATED XATTN-DENSE layers. Finish with TL;DR: CLIP vs. Flamingo.
    What to skip
    Skim Part 1 and skip Part 3 (Research Directions for LMMs), which is from 2023 and dated.

  3. Go deeper

    Visual Instruction Tuning (opens in a new tab)

    arXiv · Haotian Liu, Chunyuan Li, Qingyang Wu, Yong Jae Lee · 45 min · Free · checked 21 Oct 2026

    Why
    LLaVA is the reference design people mean by “vision encoder plus projector,” and its two-stage training recipe is a common follow-up question.
    What to read
    Read 4.1 Architecture and 4.2 Training: first pretraining the projector, then end-to-end instruction tuning. Skim section 3 on GPT-assisted visual instruction data generation.
    What to skip
    Skip section 5 Experiments and the appendices.

Companies that test Transformers most

Put Multimodal and vision-language models in your plan

Tell us your interview date and hours a day; it lands on the right day, in the right order.

Build my plan