undefined min read
Computer Architecture 01 - Overview
Core concepts of Von Neumann architecture and how CPU, memory, and buses work together
Posts tagged "memory"
undefined min read
Distributed LLM Training 06 - Where LLM Training Memory Actually Goes
Looking only at parameter size leads to bad decisions; training memory is really a combination of parameters, gradients, optimizer state, and activations
undefined min read
Distributed LLM Training 13 - Activation Checkpointing and the Cost of Recomputation
Saving memory by recomputing activations is not a minor option; it is often a central design choice in large-scale training
undefined min read
GPU Systems 03 - Memory Hierarchy and Bandwidth
How to think about the GPU memory hierarchy and bandwidth bottlenecks
undefined min read
Linux Internals 04 - Memory Management
The concept of virtual memory and how the Linux kernel manages memory
undefined min read
PyTorch Internals 07 - Tensor Lifetime, the CUDA Caching Allocator, and Memory Reuse
PyTorch GPU memory behavior is shaped by a caching allocator, so observed memory usage is not just a story about current tensor objects