undefined min read
Distributed LLM Training 14 - What ZeRO Stage 1, 2, and 3 Each Remove
ZeRO is best understood as a staged system for removing different forms of replicated training state
Posts tagged "deepspeed"
undefined min read
Distributed LLM Training 19 - How to Read Megatron-LM and DeepSpeed Structurally
Frameworks are easier to understand when you read them as bundles of parallelization and state-management choices rather than as giant feature lists