The reviewers are not in full agreement to accept this paper, however, I think it should be accepted. Although the model in the paper is not extremely novel, some elements of its elements are and it is interesting to see that training an accurate classifier with the hard attention mechanism is feasible even for ImageNet. Having said that, I agree with some of the critical comments in the reviews (e.g. necessity of comparisons with regard to the number of parameters and computational cost) and I encourage the authors to incorporate them into the camera-ready version of the paper.