I recommend this paper to be accepted. The proposed method is simple and effective. Although I disagree with some specific choices that the authors made (e.g. the choice of the attribution method, it would be a lot more interesting to use masking methods e.g. Dabkowski and Gal 2017 or Zolna, Geras and Cho 2020, which are a lot stronger), I think the general idea of fixing the attribution in the compressed network is sufficiently interesting that this paper could be appreciated at NeurIPS. Having said that, I strongly encourage the authors to take into account the input of the reviewers and improve the camera-ready version of the paper.