GPU Systems
From GPU architecture and CUDA kernels to Triton and real kernel optimization work
Engineers who want to understand how GPUs actually execute work and eventually write and optimize their own kernels.
Jae's Tech Blog
Thoughts on code, technology, and everything in between
Technical notes by Jae
Systems writing for engineers who want the deeper model.
Long-form posts on platform engineering, Linux, compilers, MLOps, and computer architecture, written to help you build stronger intuition instead of just memorize terms.
Start here
Featured series
A few strong entry points if you are new here.
Browse recent posts
119 posts
Fresh writing, updates, and ongoing series entries.
MLOps Fundamentals
Building reliable ML systems from data pipelines to production monitoring
ML engineers, data scientists, and backend engineers moving from model experiments to production operations.
Automata and Compilers
From finite automata and formal languages to building a compiler from scratch
Readers who want both the theory behind language processing and the bridge to real compiler construction.
UPDATED
- GPU Systems 20 - From Nsight to Triton to FlashAttention NEW
- GPU Systems 19 - Asynchronous Copy and Pipelining
- GPU Systems 18 - Tensor Cores and Mixed Precision
- GPU Systems 17 - Register Pressure and Spilling
- GPU Systems 16 - Vectorized Loads, Stores, and Alignment
- GPU Systems 15 - LayerNorm and RMSNorm Kernel Structure
- GPU Systems 14 - Why Softmax Is Such a Good Kernel Exercise
- GPU Systems 13 - Reduction Kernels in Depth
- GPU Systems 12 - Warp Shuffle and Warp-Level Primitives
- GPU Systems 11 - Shared Memory Bank Conflicts
- GPU Systems 10 - Tiled Matrix Multiplication and Shared Memory
- GPU Systems 09 - Why Naive Matrix Multiplication Is Slow
- GPU Systems 08 - Profiling and the Roofline View
- GPU Systems 07 - Occupancy and Latency Hiding
- GPU Systems 06 - Triton and the Practical Shape of Kernel Optimization
- GPU Systems 05 - Coalescing, Shared Memory, and Reduction Patterns
- GPU Systems 04 - Writing CUDA Kernels and Choosing Launch Configuration
- GPU Systems 03 - Memory Hierarchy and Bandwidth
- GPU Systems 02 - The Thread, Warp, and Block Execution Model
- GPU Systems 01 - Roadmap to GPU Kernel Engineering
- GPU Systems 00 - What You Should Know Before Starting This Series
- MLOps 10 - Building an MLOps Platform
- MLOps 09 - GPU Infrastructure and Scaling
- MLOps 08 - Feature Stores
- MLOps 07 - CI/CD for ML
- MLOps 06 - Monitoring and Drift Detection
- MLOps 05 - Model Serving and Deployment Strategies
- MLOps 04 - Model Versioning and Registry
- MLOps 03 - Experiment Tracking and Training Management
- MLOps 02 - Data Pipelines and Feature Engineering
- MLOps 01 - What Is MLOps?
Automata and Compilers
From finite automata and formal languages to building a compiler from scratch
- Automata and Compilers 12 - Code Generation
- Automata and Compilers 11 - Intermediate Representations and Optimization
- Automata and Compilers 10 - Semantic Analysis and Type Checking
- Automata and Compilers 09 - Abstract Syntax Trees
- Automata and Compilers 08 - Bottom-Up Parsing
- Automata and Compilers 07 - Top-Down Parsing
- Automata and Compilers 06 - Lexical Analysis
- Automata and Compilers 05 - Compiler Overview โ Phases and Architecture
- Automata and Compilers 04 - Pushdown Automata
- Automata and Compilers 03 - Context-Free Grammars
- Automata and Compilers 02 - Regular Expressions and Regular Languages
- Automata and Compilers 01 - Finite Automata
- Linux Internals 10 - Containers and Virtualization
- Linux Internals 09 - Networking
- Linux Internals 08 - Synchronization and Concurrency
- Linux Internals 07 - I/O and Devices
- Linux Internals 06 - System Calls and the Kernel
- Linux Internals 05 - File Systems
- Linux Internals 04 - Memory Management
- Linux Internals 03 - Process Scheduling
- Linux Internals 02 - Processes and Threads
- Linux Internals 01 - Operating System Overview
- Computer Architecture 10 - Multicore and Modern Processors
- Computer Architecture 09 - I/O and DMA
- Computer Architecture 08 - Virtual Memory and MMU
- Computer Architecture 07 - Memory Hierarchy
- Computer Architecture 06 - Interrupts and Exceptions
- Computer Architecture 05 - CPU Privilege Levels and Protection
- Computer Architecture 04 - Pipelining and Parallel Processing
- Computer Architecture 03 - Instruction Set Architecture (ISA)
- Computer Architecture 02 - CPU Internals
- Computer Architecture 01 - Overview
- Platform Engineering 11 - Building a Platform Team
- Platform Engineering 10 - Security and Governance
- Platform Engineering 09 - Observability for Platform Engineers
- Platform Engineering 08 - CI/CD at Scale
- Platform Engineering 07 - Platform as a Product
- Platform Engineering 06 - Developer Portals and Service Catalogs
- Platform Engineering 05 - Infrastructure as Code for Platforms
- Platform Engineering 04 - Designing Golden Paths
- Platform Engineering 03 - Developer Experience as a North Star
- Platform Engineering 02 - Anatomy of an Internal Developer Platform
- Platform Engineering 01 - What Is Platform Engineering?
- Distributed LLM Training 20 - A Practical Order for Designing an LLM Training Stack
- Distributed LLM Training 19 - How to Read Megatron-LM and DeepSpeed Structurally
- Distributed LLM Training 18 - Deadlocks, Timeouts, and OOMs: Debugging Distributed Training
- Distributed LLM Training 17 - Why Checkpointing, Resume, and Fault Tolerance Matter So Much
- Distributed LLM Training 16 - How Communication Overlap Hides Step Time
- Distributed LLM Training 15 - How FSDP Differs from DDP and When It Helps
- Distributed LLM Training 14 - What ZeRO Stage 1, 2, and 3 Each Remove
- Distributed LLM Training 13 - Activation Checkpointing and the Cost of Recomputation
- Distributed LLM Training 12 - GPipe, 1F1B, and Interleaving: Choosing a Pipeline Schedule
- Distributed LLM Training 11 - Pipeline Parallel Basics and How to Think About Stage Splits
- Distributed LLM Training 10 - Sequence Parallelism and the Cost of Long Context
- Distributed LLM Training 09 - Where Tensor Parallelism Actually Lives Inside a Transformer
- Distributed LLM Training 08 - Tensor Parallel Basics: Splitting Computation Inside the Model
- Distributed LLM Training 07 - NCCL and Topology: Why the Same GPU Count Can Behave Very Differently
- Distributed LLM Training 06 - Where LLM Training Memory Actually Goes
- Distributed LLM Training 05 - Global Batch Size, Gradient Accumulation, and Learning Rate Scaling
- Distributed LLM Training 04 - What PyTorch DDP Actually Does Internally
- Distributed LLM Training 03 - All-Reduce, Ring, and How to Read Communication Cost
- Distributed LLM Training 02 - The Real Cost of Synchronous SGD and Data Parallelism
- Distributed LLM Training 01 - Why LLM Training Becomes a Distributed Systems Problem
- PyTorch Internals 20 - A Practical Path from Internals Knowledge to Real Engineering Work
- PyTorch Internals 19 - Extension Packaging, Testing, and ABI Stability
- PyTorch Internals 18 - Where Autograd Meets Distributed Runtime
- PyTorch Internals 17 - What Role Triton Plays Inside the PyTorch Ecosystem
- PyTorch Internals 16 - The Big Picture of FX, torch.compile, and Inductor
- PyTorch Internals 15 - Reading Operator Bottlenecks with PyTorch Profiling
- PyTorch Internals 14 - AMP, Autocast, and Numerical Stability
- PyTorch Internals 13 - When a Fused Operator Is Actually Worth It
- PyTorch Internals 12 - Backward Implementation Patterns and Saved-State Strategy
- PyTorch Internals 11 - Operator Schema, Dispatch Keys, and Meta Functions
- PyTorch Internals 10 - Connecting a Custom CUDA Kernel Through an Extension
- PyTorch Internals 09 - The Basic Path of a C++ Extension
- PyTorch Internals 08 - CUDA Streams, Events, and Asynchronous Execution
- PyTorch Internals 07 - Tensor Lifetime, the CUDA Caching Allocator, and Memory Reuse
- PyTorch Internals 06 - When and How to Use a Custom Autograd Function
- PyTorch Internals 05 - How the Autograd Graph and Engine Work
- PyTorch Internals 04 - What the Dispatcher and Operator Registry Actually Do
- PyTorch Internals 03 - Contiguous Layout, Memory Format, and Hidden Copies
- PyTorch Internals 02 - Tensors Run on Top of Storage, Size, and Stride
- PyTorch Internals 01 - Why You Need to Understand the Internals
undefined min read
Distributed LLM Training 10 - Sequence Parallelism and the Cost of Long Context
As context length grows, activation memory and communication patterns change again, and sequence-oriented partitioning starts to matter
undefined min read
GPU Systems 02 - The Thread, Warp, and Block Execution Model
What threads, warps, blocks, and grids mean in actual GPU execution
undefined min read
PyTorch Internals 10 - Connecting a Custom CUDA Kernel Through an Extension
A CUDA kernel becomes a real PyTorch operator only when tensor contracts, runtime semantics, and integration details are handled correctly
undefined min read
Platform Engineering 06 - Developer Portals and Service Catalogs
How developer portals like Backstage bring order to the chaos of scattered docs, tools, and tribal knowledge
undefined min read
Distributed LLM Training 09 - Where Tensor Parallelism Actually Lives Inside a Transformer
Tensor parallelism becomes real when you map it onto QKV projections, attention output paths, and the two large MLP projections inside a transformer block
undefined min read
GPU Systems 01 - Roadmap to GPU Kernel Engineering
A practical study order from GPU architecture to CUDA, Triton, and kernel optimization
undefined min read
MLOps 05 - Model Serving and Deployment Strategies
How to serve trained models in production and deploy them safely
undefined min read
PyTorch Internals 09 - The Basic Path of a C++ Extension
A C++ extension is the first practical bridge between user-defined logic and the PyTorch runtime
undefined min read
GPU Systems 00 - What You Should Know Before Starting This Series
The background knowledge that makes the GPU Systems series much easier to study properly