Jae's Tech Blog

Thoughts on code, technology, and everything in between

Technical notes by Jae

Systems writing for engineers who want the deeper model.

Long-form posts on platform engineering, Linux, compilers, MLOps, and computer architecture, written to help you build stronger intuition instead of just memorize terms.

Start here Featured series

A few strong entry points if you are new here.

Browse recent posts 119 posts

Fresh writing, updates, and ongoing series entries.

21 posts 58 min total

GPU Systems

From GPU architecture and CUDA kernels to Triton and real kernel optimization work

Engineers who want to understand how GPUs actually execute work and eventually write and optimize their own kernels.

Recommended Start
10 posts 43 min total

MLOps Fundamentals

Building reliable ML systems from data pipelines to production monitoring

ML engineers, data scientists, and backend engineers moving from model experiments to production operations.

Recommended Start
12 posts 60 min total

Automata and Compilers

From finite automata and formal languages to building a compiler from scratch

Readers who want both the theory behind language processing and the bridge to real compiler construction.

Recommended Start
UPDATED

GPU Systems

From GPU architecture and CUDA kernels to Triton and real kernel optimization work

21 posts
Open series guide โ†’
  1. GPU Systems 20 - From Nsight to Triton to FlashAttention NEW
  2. GPU Systems 19 - Asynchronous Copy and Pipelining
  3. GPU Systems 18 - Tensor Cores and Mixed Precision
  4. GPU Systems 17 - Register Pressure and Spilling
  5. GPU Systems 16 - Vectorized Loads, Stores, and Alignment
  6. GPU Systems 15 - LayerNorm and RMSNorm Kernel Structure
  7. GPU Systems 14 - Why Softmax Is Such a Good Kernel Exercise
  8. GPU Systems 13 - Reduction Kernels in Depth
  9. GPU Systems 12 - Warp Shuffle and Warp-Level Primitives
  10. GPU Systems 11 - Shared Memory Bank Conflicts
  11. GPU Systems 10 - Tiled Matrix Multiplication and Shared Memory
  12. GPU Systems 09 - Why Naive Matrix Multiplication Is Slow
  13. GPU Systems 08 - Profiling and the Roofline View
  14. GPU Systems 07 - Occupancy and Latency Hiding
  15. GPU Systems 06 - Triton and the Practical Shape of Kernel Optimization
  16. GPU Systems 05 - Coalescing, Shared Memory, and Reduction Patterns
  17. GPU Systems 04 - Writing CUDA Kernels and Choosing Launch Configuration
  18. GPU Systems 03 - Memory Hierarchy and Bandwidth
  19. GPU Systems 02 - The Thread, Warp, and Block Execution Model
  20. GPU Systems 01 - Roadmap to GPU Kernel Engineering
  21. GPU Systems 00 - What You Should Know Before Starting This Series

MLOps Fundamentals

Building reliable ML systems from data pipelines to production monitoring

10 posts
Open series guide โ†’
  1. MLOps 10 - Building an MLOps Platform
  2. MLOps 09 - GPU Infrastructure and Scaling
  3. MLOps 08 - Feature Stores
  4. MLOps 07 - CI/CD for ML
  5. MLOps 06 - Monitoring and Drift Detection
  6. MLOps 05 - Model Serving and Deployment Strategies
  7. MLOps 04 - Model Versioning and Registry
  8. MLOps 03 - Experiment Tracking and Training Management
  9. MLOps 02 - Data Pipelines and Feature Engineering
  10. MLOps 01 - What Is MLOps?

Automata and Compilers

From finite automata and formal languages to building a compiler from scratch

12 posts
Open series guide โ†’
  1. Automata and Compilers 12 - Code Generation
  2. Automata and Compilers 11 - Intermediate Representations and Optimization
  3. Automata and Compilers 10 - Semantic Analysis and Type Checking
  4. Automata and Compilers 09 - Abstract Syntax Trees
  5. Automata and Compilers 08 - Bottom-Up Parsing
  6. Automata and Compilers 07 - Top-Down Parsing
  7. Automata and Compilers 06 - Lexical Analysis
  8. Automata and Compilers 05 - Compiler Overview โ€” Phases and Architecture
  9. Automata and Compilers 04 - Pushdown Automata
  10. Automata and Compilers 03 - Context-Free Grammars
  11. Automata and Compilers 02 - Regular Expressions and Regular Languages
  12. Automata and Compilers 01 - Finite Automata

Linux Internals

Understanding the Linux kernel from processes and memory to containers

10 posts
Open series guide โ†’
  1. Linux Internals 10 - Containers and Virtualization
  2. Linux Internals 09 - Networking
  3. Linux Internals 08 - Synchronization and Concurrency
  4. Linux Internals 07 - I/O and Devices
  5. Linux Internals 06 - System Calls and the Kernel
  6. Linux Internals 05 - File Systems
  7. Linux Internals 04 - Memory Management
  8. Linux Internals 03 - Process Scheduling
  9. Linux Internals 02 - Processes and Threads
  10. Linux Internals 01 - Operating System Overview

Computer Architecture

From CPU internals and privilege levels to memory hierarchy and modern multicore processors

10 posts
Open series guide โ†’
  1. Computer Architecture 10 - Multicore and Modern Processors
  2. Computer Architecture 09 - I/O and DMA
  3. Computer Architecture 08 - Virtual Memory and MMU
  4. Computer Architecture 07 - Memory Hierarchy
  5. Computer Architecture 06 - Interrupts and Exceptions
  6. Computer Architecture 05 - CPU Privilege Levels and Protection
  7. Computer Architecture 04 - Pipelining and Parallel Processing
  8. Computer Architecture 03 - Instruction Set Architecture (ISA)
  9. Computer Architecture 02 - CPU Internals
  10. Computer Architecture 01 - Overview

Platform Engineering Fundamentals

Understanding the principles behind Internal Developer Platforms, golden paths, and developer self-service

11 posts
Open series guide โ†’
  1. Platform Engineering 11 - Building a Platform Team
  2. Platform Engineering 10 - Security and Governance
  3. Platform Engineering 09 - Observability for Platform Engineers
  4. Platform Engineering 08 - CI/CD at Scale
  5. Platform Engineering 07 - Platform as a Product
  6. Platform Engineering 06 - Developer Portals and Service Catalogs
  7. Platform Engineering 05 - Infrastructure as Code for Platforms
  8. Platform Engineering 04 - Designing Golden Paths
  9. Platform Engineering 03 - Developer Experience as a North Star
  10. Platform Engineering 02 - Anatomy of an Internal Developer Platform
  11. Platform Engineering 01 - What Is Platform Engineering?

Distributed LLM Training

From data parallelism to tensor parallelism, FSDP, ZeRO, and modern LLM training frameworks

20 posts
Open series guide โ†’
  1. Distributed LLM Training 20 - A Practical Order for Designing an LLM Training Stack
  2. Distributed LLM Training 19 - How to Read Megatron-LM and DeepSpeed Structurally
  3. Distributed LLM Training 18 - Deadlocks, Timeouts, and OOMs: Debugging Distributed Training
  4. Distributed LLM Training 17 - Why Checkpointing, Resume, and Fault Tolerance Matter So Much
  5. Distributed LLM Training 16 - How Communication Overlap Hides Step Time
  6. Distributed LLM Training 15 - How FSDP Differs from DDP and When It Helps
  7. Distributed LLM Training 14 - What ZeRO Stage 1, 2, and 3 Each Remove
  8. Distributed LLM Training 13 - Activation Checkpointing and the Cost of Recomputation
  9. Distributed LLM Training 12 - GPipe, 1F1B, and Interleaving: Choosing a Pipeline Schedule
  10. Distributed LLM Training 11 - Pipeline Parallel Basics and How to Think About Stage Splits
  11. Distributed LLM Training 10 - Sequence Parallelism and the Cost of Long Context
  12. Distributed LLM Training 09 - Where Tensor Parallelism Actually Lives Inside a Transformer
  13. Distributed LLM Training 08 - Tensor Parallel Basics: Splitting Computation Inside the Model
  14. Distributed LLM Training 07 - NCCL and Topology: Why the Same GPU Count Can Behave Very Differently
  15. Distributed LLM Training 06 - Where LLM Training Memory Actually Goes
  16. Distributed LLM Training 05 - Global Batch Size, Gradient Accumulation, and Learning Rate Scaling
  17. Distributed LLM Training 04 - What PyTorch DDP Actually Does Internally
  18. Distributed LLM Training 03 - All-Reduce, Ring, and How to Read Communication Cost
  19. Distributed LLM Training 02 - The Real Cost of Synchronous SGD and Data Parallelism
  20. Distributed LLM Training 01 - Why LLM Training Becomes a Distributed Systems Problem

PyTorch Internals

Understanding tensors, autograd, and CUDA extensions well enough to connect custom kernels to real training code

20 posts
Open series guide โ†’
  1. PyTorch Internals 20 - A Practical Path from Internals Knowledge to Real Engineering Work
  2. PyTorch Internals 19 - Extension Packaging, Testing, and ABI Stability
  3. PyTorch Internals 18 - Where Autograd Meets Distributed Runtime
  4. PyTorch Internals 17 - What Role Triton Plays Inside the PyTorch Ecosystem
  5. PyTorch Internals 16 - The Big Picture of FX, torch.compile, and Inductor
  6. PyTorch Internals 15 - Reading Operator Bottlenecks with PyTorch Profiling
  7. PyTorch Internals 14 - AMP, Autocast, and Numerical Stability
  8. PyTorch Internals 13 - When a Fused Operator Is Actually Worth It
  9. PyTorch Internals 12 - Backward Implementation Patterns and Saved-State Strategy
  10. PyTorch Internals 11 - Operator Schema, Dispatch Keys, and Meta Functions
  11. PyTorch Internals 10 - Connecting a Custom CUDA Kernel Through an Extension
  12. PyTorch Internals 09 - The Basic Path of a C++ Extension
  13. PyTorch Internals 08 - CUDA Streams, Events, and Asynchronous Execution
  14. PyTorch Internals 07 - Tensor Lifetime, the CUDA Caching Allocator, and Memory Reuse
  15. PyTorch Internals 06 - When and How to Use a Custom Autograd Function
  16. PyTorch Internals 05 - How the Autograd Graph and Engine Work
  17. PyTorch Internals 04 - What the Dispatcher and Operator Registry Actually Do
  18. PyTorch Internals 03 - Contiguous Layout, Memory Format, and Hidden Copies
  19. PyTorch Internals 02 - Tensors Run on Top of Storage, Size, and Stride
  20. PyTorch Internals 01 - Why You Need to Understand the Internals

Python Lecture Series

Comprehensive Python programming course from basics to advanced topics

5 posts
Open series guide โ†’
  1. Python Lecture 04 - Class Internals and Descriptors
  2. Python Lecture 3 - Containers and Generators
  3. Python Lecture 2 - Lexical Structure of Python
  4. Python Lecture Part 1 - Python's Philosophy and Design Principles
  5. Introduction to Python Lecture Series