Skip to content

GPU programming basics: CUDA and Triton kernels

  • Inference
  • Specialist topic or differentiator
  • About 2 h 30 min to learn
  • 3 vetted resources
Build a plan that includes this

What this topic covers

Inference and performance roles at labs, chip companies and serving startups list CUDA or Triton in the job description, and their interviews can ask you to write or reason about a simple kernel. Even outside those roles, knowing why a fused kernel beats three PyTorch calls explains FlashAttention, quantized GEMMs and most serving speed-ups. A strong answer covers the thread, block and grid model, memory hierarchy and coalesced access, why most LLM ops are memory-bound so fusion pays off, how Triton lets you write block-level kernels in Python, and how you’d profile and benchmark against the PyTorch baseline.

Builds on

You’re ready when you can

What interviewers check, from the resources below.

  • Explain why the naive PyTorch softmax is memory-bound
  • Write a Triton kernel where each program instance handles one row
  • Explain the role of BLOCK_SIZE and masking for rows that aren’t a power of two
  • Read the bandwidth benchmark and say how close you are to the hardware limit
  • Compute a global thread index from block and thread ids
  • Write a grid-stride loop and explain why it’s useful
  • Explain how blocks map onto streaming multiprocessors
  • Profile a kernel and explain where the time went
  • Load and store a block with correct masks
  • Write a tiled matmul and explain the tiling choice

What to study, in order

Each link opened and checked by hand; the date says when.

  1. Start here

    Fused Softmax (opens in a new tab)

    Triton documentation · 60 min · Free · checked 22 Oct 2026

    Why
    Writing a fused softmax is the “hello world” of kernel interviews, and this tutorial shows exactly why fusion wins: one read from memory instead of several.
    What to read
    Do tutorial 01 (vector addition) first if Triton is new, then read “Motivations”, write and run the “Compute Kernel”, run the “Unit Test” and reproduce the “Benchmark” against torch.softmax on a GPU (Colab is fine).
    What to skip
    Skip the later tutorials (matmul, layer norm, attention) until this one makes sense.

  2. Free alternative

    An Even Easier Introduction to CUDA (Updated) (opens in a new tab)

    NVIDIA Technical Blog · Mark Harris · 40 min · Free · checked 22 Oct 2026

    Why
    The standard first CUDA program, updated in 2025. It gives you the thread, block and grid vocabulary every kernel question assumes.
    What to read
    Work through “Starting Simple”, “Memory Allocation in CUDA”, “Profile it!”, “Picking up the Threads”, “Out of the Blocks” and “Unified Memory Prefetching”, then do the exercises.
    What to skip
    Skip “Where To From Here?” links for now.

  3. Go deeper

    Triton-Puzzles (opens in a new tab)

    GitHub · gpu-mode · 90 min · Free · checked 22 Oct 2026

    Why
    A graded set of kernel exercises that build from trivial loads up to a FlashAttention-style kernel and quantized matmul. It runs on a Triton interpreter, so no GPU is needed.
    What to read
    Open Triton-Puzzles.ipynb in Colab from the README badge and solve Puzzles 1–5 (constant add through fused outer multiplication), then 7 (Long Sum), 8 (Long Softmax), 9 (Simple FlashAttention) and 11 (Matrix Multiplication). Explain each solution out loud as you would in an interview.
    What to skip
    Puzzles 6 (backwards), 10 (2D convolution) and 12 (quantized matmul) are optional for inference interviews.

Companies that test Inference most

Put GPU programming basics in your plan

Tell us your interview date and hours a day; it lands on the right day, in the right order.

Build my plan