NeurIPS 2020

AutoSync: Learning to Synchronize for Data-Parallel Distributed Deep Learning

Meta Review

The authors cast the task of parallel training as a learning problem, allowing data driven decisions to be made instead of the hand-crafted rules. The topic is relevant and the results are impactful. The comprehensive ablation studies performed to evaluate the system are also appreciated. Several aspects of the proposed system have room for improvement, both in terms of scope and quality. However, that doesn’t seem to be a crucial problem with the paper but rather room for follow up works.