undefined min read
Distributed LLM Training 16 - How Communication Overlap Hides Step Time
The goal of overlap is not to eliminate communication entirely, but to make it finish underneath useful computation
Posts tagged "performance"
undefined min read
GPU Systems 07 - Occupancy and Latency Hiding
Understanding occupancy as a latency-hiding concept instead of just a percentage
undefined min read
GPU Systems 08 - Profiling and the Roofline View
A practical way to use profiling and roofline thinking to understand kernel bottlenecks
undefined min read
GPU Systems 09 - Why Naive Matrix Multiplication Is Slow
Using naive matrix multiplication to see memory reuse and traffic problems clearly
undefined min read
GPU Systems 18 - Tensor Cores and Mixed Precision
How tensor cores change performance in compute-heavy kernels and why mixed precision matters
undefined min read
PyTorch Internals 03 - Contiguous Layout, Memory Format, and Hidden Copies
Layout affects both operator selection and performance, and sometimes the most expensive thing in a path is an invisible copy
undefined min read
PyTorch Internals 13 - When a Fused Operator Is Actually Worth It
Fusion is valuable when it reduces memory traffic and intermediate materialization, not just when it reduces the number of visible ops
undefined min read
PyTorch Internals 15 - Reading Operator Bottlenecks with PyTorch Profiling
The purpose of internals knowledge is to make a performance trace interpretable enough that you can actually change it