This paper introduces a novel method for neural architecture search motivated by the classic idea of gradient boosting for weak learners. The reviewers highlight the elegance of this idea in how it lifts an old idea into an area of contemporary interest. I thus recommend this paper for acceptance.