undefined min read
GPU Systems 11 - Shared Memory Bank Conflicts
Why shared memory is not automatically fast and how bank conflicts appear
Posts tagged "cuda"
undefined min read
GPU Systems 12 - Warp Shuffle and Warp-Level Primitives
Why warp-level primitives matter for reductions and lighter-weight cooperation
undefined min read
GPU Systems 13 - Reduction Kernels in Depth
Using reduction kernels to connect shared memory, warp primitives, and synchronization
undefined min read
GPU Systems 14 - Why Softmax Is Such a Good Kernel Exercise
How softmax combines reductions, memory traffic, and numerical stability in one kernel
undefined min read
GPU Systems 15 - LayerNorm and RMSNorm Kernel Structure
Why normalization kernels are often memory-bound and structurally important
undefined min read
GPU Systems 16 - Vectorized Loads, Stores, and Alignment
How wider memory operations and alignment affect bandwidth utilization
undefined min read
GPU Systems 17 - Register Pressure and Spilling
Why using more registers can improve local efficiency but still reduce total throughput
undefined min read
GPU Systems 19 - Asynchronous Copy and Pipelining
How asynchronous copy and double buffering help overlap memory movement with computation
undefined min read
PyTorch Internals 07 - Tensor Lifetime, the CUDA Caching Allocator, and Memory Reuse
PyTorch GPU memory behavior is shaped by a caching allocator, so observed memory usage is not just a story about current tensor objects