undefined min read
GPU Systems 12 - Warp Shuffle and Warp-Level Primitives
Why warp-level primitives matter for reductions and lighter-weight cooperation
Lectures
All posts in the Lectures
undefined min read
GPU Systems 13 - Reduction Kernels in Depth
Using reduction kernels to connect shared memory, warp primitives, and synchronization
undefined min read
GPU Systems 14 - Why Softmax Is Such a Good Kernel Exercise
How softmax combines reductions, memory traffic, and numerical stability in one kernel
undefined min read
GPU Systems 15 - LayerNorm and RMSNorm Kernel Structure
Why normalization kernels are often memory-bound and structurally important
undefined min read
GPU Systems 16 - Vectorized Loads, Stores, and Alignment
How wider memory operations and alignment affect bandwidth utilization
undefined min read
GPU Systems 17 - Register Pressure and Spilling
Why using more registers can improve local efficiency but still reduce total throughput
undefined min read
GPU Systems 18 - Tensor Cores and Mixed Precision
How tensor cores change performance in compute-heavy kernels and why mixed precision matters
undefined min read
GPU Systems 19 - Asynchronous Copy and Pipelining
How asynchronous copy and double buffering help overlap memory movement with computation
undefined min read
GPU Systems 20 - From Nsight to Triton to FlashAttention
Closing the GPU Systems series by connecting profiling, Triton experimentation, and FlashAttention-style thinking