NeurIPS 2020

Learning Black-Box Attackers with Transferable Priors and Query Feedback

Meta Review

This paper introduces a method to generate black-box adversarial examples by relying both on the transferability of adversarial examples, and a query-based mechanism. The reviewers were conflicted on the paper. All reviewers agreed on the strengths. The paper presents impressive results, and the attack is significantly stronger than the SimBA baseline from prior work. Reviewers also agreed on the differences, but diverged in their thoughts on if it was important. This paper is, in many ways, a collection of tricks that helps to improve the efficacy of SimBA. The authors dispute this claim in the response, but I agree with the reviewers. In both the abstract and the rebuttal, the authors argue that the scheme is simple and the techniques generalize. However, there is not any evidence that this is the case in the main body of the paper, and the supplied code does not justify this claim. Nevertheless, the strengths do outweigh the weaknesses. The strong results are important even if the paper does not (yet) show that the results will transfer. The strong results will be an important baseline for future work.