Skip to content

Vector database internals and index selection

  • RAG
  • Specialist topic or differentiator
  • About 1 h 50 min to learn
  • 4 vetted resources
Build a plan that includes this

What this topic covers

Approximate nearest-neighbour search trades recall for speed and memory, and the index you pick decides that trade. HNSW gives high recall with a graph that costs memory and slow inserts; IVF clusters and probes a few cells; product quantization compresses vectors so they fit in RAM at the cost of recall. Interviewers then push on the practical parts: how metadata filters interact with an ANN index (pre- vs post-filtering), hybrid keyword plus vector ranking, what an update really does to the index, and the back-of-envelope maths for sizing a corpus. The classic design-round decision is pgvector inside Postgres versus a dedicated vector database, and you should be able to defend either with numbers.

Builds on

You’re ready when you can

What interviewers check, from the resources below.

  • Compare HNSW and IVF on recall, memory, build time and insert cost
  • Explain why post-filtering can return fewer than k results and when pre-filtering is the better choice
  • Compute storage for 10M vectors at 1536 dimensions in float32, with and without HNSW overhead
  • Describe how updates are handled (delete plus insert, hot/cold index) and what consistency you promise
  • Argue when to stay on pgvector and when to move to a dedicated vector database
  • Explain what m, ef_construction and ef_search each control and which direction raises recall
  • Give the rule of thumb for IVFFlat lists and probes and why the index should be built after loading data
  • Reproduce the README's argument for why a 10%-selective WHERE clause returns about 4 rows with default settings
  • Describe iterative index scans and the difference between strict and relaxed ordering
  • Write a hybrid query combining vector distance with full-text search and merge the results with RRF

What to study, in order

Each link opened and checked by hand; the date says when.

  1. Start here

    Vector Databases Deep Dive for System Design Interviews (opens in a new tab)

    Hello Interview · Hello Interview · 45 min · Free · checked 21 Oct 2026

    Why
    It is written for exactly the question you will get ('design semantic search over N documents') and gives you the index comparison, the filtering trap, and the sizing numbers in one place.
    What to read
    Read “How Vector Databases Work” (all of “Indexing Strategies”, “Filtering and Hybrid Search”, “Inserts, Updates, and Index Maintenance”), “Vector Database Options”, and “Using Vector Databases in Your Interview” through “Numbers to Know”.
    What to skip
    “What's a Vector Anyway?” and “Similarity Metrics” if you have done the embeddings topic; LSH and Annoy are rarely asked.

  2. Free alternative

    pgvector: open-source vector similarity search for Postgres (opens in a new tab)

    GitHub · Andrew Kane · 30 min · Free · checked 21 Oct 2026

    Why
    Interviewers love 'why not just Postgres?', and the README is the honest answer: real parameters, the filtering problem spelled out with numbers, and hybrid search in plain SQL.
    What to read
    Read “Indexing”, “HNSW”, “IVFFlat”, “Filtering”, “Iterative Index Scans”, “Hybrid Search” and “Performance”.
    What to skip
    Installation, language bindings, half-precision/binary/sparse vector sections and the type reference.

  3. Go deeper

    Product Quantization: Compressing high-dimensional vectors by 97% (opens in a new tab)

    Pinecone Learning Center · Pinecone · 30 min · Free · checked 21 Oct 2026

    Why
    When the corpus no longer fits in RAM, the follow-up is 'what do you compress and what do you lose', and PQ is the standard answer.
    What to read
    Read “What is Quantization” (with “Why Product Quantization?”) and “How Product Quantization Works”, then the “IndexIVFPQ” part of “PQ Implementation in Faiss” for the recall/speed/memory table.
    What to skip
    The line-by-line Faiss code under “IndexPQ”; focus on the diagrams and the numbers.

  4. Go deeper

    Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs (opens in a new tab)

    arXiv · Yu. A. Malkov et al. · 45 min · Free · checked 22 Oct 2026

    Why
    HNSW is the default index in most vector databases, and interviewers ask how its layers work and what M, efConstruction and ef trade off.
    What to read
    Read §3 on the motivation (layered graphs as a probabilistic skip list), then §4 with Algorithms 1–5 (insert, search a layer, the two neighbour-selection methods, k-NN search) and Figure 1. Then read §4.1 on how the construction parameters affect recall and speed, and §5.4 on how it compares with product quantisation.
    What to skip
    Skip §2 related work, §4.2 complexity proofs and §5.1–5.3 benchmark detail beyond the main plots.

Companies that test RAG most

Put Vector database internals and index selection in your plan

Tell us your interview date and hours a day; it lands on the right day, in the right order.

Build my plan