NeurIPS 2020

Over-parameterized Adversarial Training: An Analysis Overcoming the Curse of Dimensionality


Meta Review

This paper proves that adversarial training of over-parameterized neural networks converges to a robust solution. Specifically, the paper studies two-layer ReLU networks with width that is polynomial in the input dimension, d, the number of training points, n, and the inverse of the robustness parameter, 1/\epsilon. (This improves over prior work that required exponentially wide networks.) The proof is by construction; an algorithm is proposed that, in poly(d, n, 1/\epsilon) iterations, finds a network with poly(d, n, 1/\epsilon) width that is \epsilon-robust. Adversarial training is an important and rapidly expanding field of ML. This paper fills in some gaps w.r.t. over-parameterized neural networks (which, on their own, have recently created a cottage industry theory papers). The reviewers seem to really appreciate the paper, saying that it solves a relevant problem, and that it's clear and well written. Importantly, the authors' response appears to have cleared up a few misunderstandings from the first round of reviews. I thank the authors for writing a succinct, helpful response; and I thank the reviewers, for carefully considering what the authors wrote and incorporating this feedback into their reviews. This is a good example of the author response process working as it should. I encourage the authors' to incorporate feedback from the reviewers when revising the paper.