undefined min read
Computer Architecture 07 - Memory Hierarchy
The memory hierarchy from registers to HDD and how caches work
Lectures
All posts in the Lectures
undefined min read
Computer Architecture 08 - Virtual Memory and MMU
How virtual memory enables process isolation through the MMU, page tables, and TLB
undefined min read
Computer Architecture 09 - I/O and DMA
How the CPU exchanges data with external devices and the principles behind efficient data transfer via DMA
undefined min read
Computer Architecture 10 - Multicore and Modern Processors
Why clock speeds stopped increasing and the core concepts of modern multicore processor architecture
undefined min read
Distributed LLM Training 01 - Why LLM Training Becomes a Distributed Systems Problem
Once LLM training leaves a single GPU, it stops being only a modeling problem and becomes a systems problem around memory, communication, and recovery
undefined min read
Distributed LLM Training 02 - The Real Cost of Synchronous SGD and Data Parallelism
Data parallelism looks simple, but it carries both gradient synchronization cost and full model-state replication cost
undefined min read
Distributed LLM Training 03 - All-Reduce, Ring, and How to Read Communication Cost
To reason about distributed training performance, you need a concrete mental model for all-reduce and collective communication cost
undefined min read
Distributed LLM Training 04 - What PyTorch DDP Actually Does Internally
DDP is not just a wrapper around your model; it is a runtime that coordinates autograd hooks, gradient buckets, and synchronization timing
undefined min read
Distributed LLM Training 05 - Global Batch Size, Gradient Accumulation, and Learning Rate Scaling
Adding more GPUs changes optimizer semantics as well as throughput, so batch size and learning rate need to be reasoned about together