GPU Systems
From GPU architecture and CUDA kernels to Triton and real kernel optimization work
Engineers who want to understand how GPUs actually execute work and eventually write and optimize their own kernels.
Jae's Tech Blog
Thoughts on code, technology, and everything in between
Technical notes by Jae
Systems writing for engineers who want the deeper model.
Long-form posts on platform engineering, Linux, compilers, MLOps, and computer architecture, written to help you build stronger intuition instead of just memorize terms.
Start here
Featured series
A few strong entry points if you are new here.
Browse recent posts
119 posts
Fresh writing, updates, and ongoing series entries.
MLOps Fundamentals
Building reliable ML systems from data pipelines to production monitoring
ML engineers, data scientists, and backend engineers moving from model experiments to production operations.
Automata and Compilers
From finite automata and formal languages to building a compiler from scratch
Readers who want both the theory behind language processing and the bridge to real compiler construction.
UPDATED
- GPU Systems 20 - From Nsight to Triton to FlashAttention NEW
- GPU Systems 19 - Asynchronous Copy and Pipelining
- GPU Systems 18 - Tensor Cores and Mixed Precision
- GPU Systems 17 - Register Pressure and Spilling
- GPU Systems 16 - Vectorized Loads, Stores, and Alignment
- GPU Systems 15 - LayerNorm and RMSNorm Kernel Structure
- GPU Systems 14 - Why Softmax Is Such a Good Kernel Exercise
- GPU Systems 13 - Reduction Kernels in Depth
- GPU Systems 12 - Warp Shuffle and Warp-Level Primitives
- GPU Systems 11 - Shared Memory Bank Conflicts
- GPU Systems 10 - Tiled Matrix Multiplication and Shared Memory
- GPU Systems 09 - Why Naive Matrix Multiplication Is Slow
- GPU Systems 08 - Profiling and the Roofline View
- GPU Systems 07 - Occupancy and Latency Hiding
- GPU Systems 06 - Triton and the Practical Shape of Kernel Optimization
- GPU Systems 05 - Coalescing, Shared Memory, and Reduction Patterns
- GPU Systems 04 - Writing CUDA Kernels and Choosing Launch Configuration
- GPU Systems 03 - Memory Hierarchy and Bandwidth
- GPU Systems 02 - The Thread, Warp, and Block Execution Model
- GPU Systems 01 - Roadmap to GPU Kernel Engineering
- GPU Systems 00 - What You Should Know Before Starting This Series
- MLOps 10 - Building an MLOps Platform
- MLOps 09 - GPU Infrastructure and Scaling
- MLOps 08 - Feature Stores
- MLOps 07 - CI/CD for ML
- MLOps 06 - Monitoring and Drift Detection
- MLOps 05 - Model Serving and Deployment Strategies
- MLOps 04 - Model Versioning and Registry
- MLOps 03 - Experiment Tracking and Training Management
- MLOps 02 - Data Pipelines and Feature Engineering
- MLOps 01 - What Is MLOps?
Automata and Compilers
From finite automata and formal languages to building a compiler from scratch
- Automata and Compilers 12 - Code Generation
- Automata and Compilers 11 - Intermediate Representations and Optimization
- Automata and Compilers 10 - Semantic Analysis and Type Checking
- Automata and Compilers 09 - Abstract Syntax Trees
- Automata and Compilers 08 - Bottom-Up Parsing
- Automata and Compilers 07 - Top-Down Parsing
- Automata and Compilers 06 - Lexical Analysis
- Automata and Compilers 05 - Compiler Overview โ Phases and Architecture
- Automata and Compilers 04 - Pushdown Automata
- Automata and Compilers 03 - Context-Free Grammars
- Automata and Compilers 02 - Regular Expressions and Regular Languages
- Automata and Compilers 01 - Finite Automata
- Linux Internals 10 - Containers and Virtualization
- Linux Internals 09 - Networking
- Linux Internals 08 - Synchronization and Concurrency
- Linux Internals 07 - I/O and Devices
- Linux Internals 06 - System Calls and the Kernel
- Linux Internals 05 - File Systems
- Linux Internals 04 - Memory Management
- Linux Internals 03 - Process Scheduling
- Linux Internals 02 - Processes and Threads
- Linux Internals 01 - Operating System Overview
- Computer Architecture 10 - Multicore and Modern Processors
- Computer Architecture 09 - I/O and DMA
- Computer Architecture 08 - Virtual Memory and MMU
- Computer Architecture 07 - Memory Hierarchy
- Computer Architecture 06 - Interrupts and Exceptions
- Computer Architecture 05 - CPU Privilege Levels and Protection
- Computer Architecture 04 - Pipelining and Parallel Processing
- Computer Architecture 03 - Instruction Set Architecture (ISA)
- Computer Architecture 02 - CPU Internals
- Computer Architecture 01 - Overview
- Platform Engineering 11 - Building a Platform Team
- Platform Engineering 10 - Security and Governance
- Platform Engineering 09 - Observability for Platform Engineers
- Platform Engineering 08 - CI/CD at Scale
- Platform Engineering 07 - Platform as a Product
- Platform Engineering 06 - Developer Portals and Service Catalogs
- Platform Engineering 05 - Infrastructure as Code for Platforms
- Platform Engineering 04 - Designing Golden Paths
- Platform Engineering 03 - Developer Experience as a North Star
- Platform Engineering 02 - Anatomy of an Internal Developer Platform
- Platform Engineering 01 - What Is Platform Engineering?
- Distributed LLM Training 20 - A Practical Order for Designing an LLM Training Stack
- Distributed LLM Training 19 - How to Read Megatron-LM and DeepSpeed Structurally
- Distributed LLM Training 18 - Deadlocks, Timeouts, and OOMs: Debugging Distributed Training
- Distributed LLM Training 17 - Why Checkpointing, Resume, and Fault Tolerance Matter So Much
- Distributed LLM Training 16 - How Communication Overlap Hides Step Time
- Distributed LLM Training 15 - How FSDP Differs from DDP and When It Helps
- Distributed LLM Training 14 - What ZeRO Stage 1, 2, and 3 Each Remove
- Distributed LLM Training 13 - Activation Checkpointing and the Cost of Recomputation
- Distributed LLM Training 12 - GPipe, 1F1B, and Interleaving: Choosing a Pipeline Schedule
- Distributed LLM Training 11 - Pipeline Parallel Basics and How to Think About Stage Splits
- Distributed LLM Training 10 - Sequence Parallelism and the Cost of Long Context
- Distributed LLM Training 09 - Where Tensor Parallelism Actually Lives Inside a Transformer
- Distributed LLM Training 08 - Tensor Parallel Basics: Splitting Computation Inside the Model
- Distributed LLM Training 07 - NCCL and Topology: Why the Same GPU Count Can Behave Very Differently
- Distributed LLM Training 06 - Where LLM Training Memory Actually Goes
- Distributed LLM Training 05 - Global Batch Size, Gradient Accumulation, and Learning Rate Scaling
- Distributed LLM Training 04 - What PyTorch DDP Actually Does Internally
- Distributed LLM Training 03 - All-Reduce, Ring, and How to Read Communication Cost
- Distributed LLM Training 02 - The Real Cost of Synchronous SGD and Data Parallelism
- Distributed LLM Training 01 - Why LLM Training Becomes a Distributed Systems Problem
- PyTorch Internals 20 - A Practical Path from Internals Knowledge to Real Engineering Work
- PyTorch Internals 19 - Extension Packaging, Testing, and ABI Stability
- PyTorch Internals 18 - Where Autograd Meets Distributed Runtime
- PyTorch Internals 17 - What Role Triton Plays Inside the PyTorch Ecosystem
- PyTorch Internals 16 - The Big Picture of FX, torch.compile, and Inductor
- PyTorch Internals 15 - Reading Operator Bottlenecks with PyTorch Profiling
- PyTorch Internals 14 - AMP, Autocast, and Numerical Stability
- PyTorch Internals 13 - When a Fused Operator Is Actually Worth It
- PyTorch Internals 12 - Backward Implementation Patterns and Saved-State Strategy
- PyTorch Internals 11 - Operator Schema, Dispatch Keys, and Meta Functions
- PyTorch Internals 10 - Connecting a Custom CUDA Kernel Through an Extension
- PyTorch Internals 09 - The Basic Path of a C++ Extension
- PyTorch Internals 08 - CUDA Streams, Events, and Asynchronous Execution
- PyTorch Internals 07 - Tensor Lifetime, the CUDA Caching Allocator, and Memory Reuse
- PyTorch Internals 06 - When and How to Use a Custom Autograd Function
- PyTorch Internals 05 - How the Autograd Graph and Engine Work
- PyTorch Internals 04 - What the Dispatcher and Operator Registry Actually Do
- PyTorch Internals 03 - Contiguous Layout, Memory Format, and Hidden Copies
- PyTorch Internals 02 - Tensors Run on Top of Storage, Size, and Stride
- PyTorch Internals 01 - Why You Need to Understand the Internals
undefined min read
Automata and Compilers 06 - Lexical Analysis
How a lexer breaks source code into tokens and where automata theory meets real implementation
undefined min read
Distributed LLM Training 06 - Where LLM Training Memory Actually Goes
Looking only at parameter size leads to bad decisions; training memory is really a combination of parameters, gradients, optimizer state, and activations
undefined min read
PyTorch Internals 06 - When and How to Use a Custom Autograd Function
Custom autograd functions are a practical place to define forward-backward contracts before dropping to lower-level extensions
undefined min read
Linux Internals 04 - Memory Management
The concept of virtual memory and how the Linux kernel manages memory
undefined min read
Computer Architecture 04 - Pipelining and Parallel Processing
Instruction pipelining, hazard handling, branch prediction, superscalar and out-of-order execution
undefined min read
Distributed LLM Training 05 - Global Batch Size, Gradient Accumulation, and Learning Rate Scaling
Adding more GPUs changes optimizer semantics as well as throughput, so batch size and learning rate need to be reasoned about together
undefined min read
Platform Engineering 04 - Designing Golden Paths
How to design golden paths that developers actually want to follow โ principles, step-by-step process, and handling teams that go off-road
undefined min read
PyTorch Internals 05 - How the Autograd Graph and Engine Work
Autograd is not just automatic differentiation; it is a graph-construction and backward-execution runtime
undefined min read
Distributed LLM Training 04 - What PyTorch DDP Actually Does Internally
DDP is not just a wrapper around your model; it is a runtime that coordinates autograd hooks, gradient buckets, and synchronization timing