undefined min read
Distributed LLM Training 14 - What ZeRO Stage 1, 2, and 3 Each Remove
ZeRO is best understood as a staged system for removing different forms of replicated training state
Posts tagged "sharding"
undefined min read
Distributed LLM Training 15 - How FSDP Differs from DDP and When It Helps
FSDP keeps parameters sharded and only gathers them when needed, making it a direct answer to parameter-replication pressure