NeurIPS 2019
Sun Dec 8th through Sat the 14th, 2019 at Vancouver Convention Center
Paper ID:328
Title:Towards closing the gap between the theory and practice of SVRG

All the reviewers agreed that this is a nice paper that gives some theoretical grounding for some of the empirical choices made when applying SVRG in practice. In particular, they analyze smaller epochs (m=n) and non-iid sampling/batching (e.g. minibatch without replacement), and they analyze an algorithm where the first iterate of an epoch is the last iterate of the previous one rather than an average over the previous epoch. While individually, these contributions are somewhat modest, together they paint a fairly complete picture of SVRG that will be useful for the community.