The authors propose a very interesting technique that is all-reduce compatible, for communication efficient learning. During the rebuttal phase the authors addressed most of the comments raised by the reviewers. The authors are strongly encouraged to address before the camera ready: - reporting end-to-end speedup of PowerSGD - adding a comparison of speedup curves comparing to various baselines on converging to certain accuracies.