undefined min read
Distributed LLM Training 16 - How Communication Overlap Hides Step Time
The goal of overlap is not to eliminate communication entirely, but to make it finish underneath useful computation
The goal of overlap is not to eliminate communication entirely, but to make it finish underneath useful computation