After much discussion, the reviewers largely converged towards recommending to accept this submission. The reviewers appreciate the merits of the paper, believe it investigates important open questions, and will thus be a significant contribution to our understanding of BNNs, but only when the experimental issues mentioned in the reviews are resolved. I would draw the author's attention to the fact that the reviewers raised concerns about the supplementary material containing a number of sections which are not connected to results in the main paper (on tempered posteriors, sampling from the prior, discussions of what’s Bayesian, PAC Bayes etc.). Per reviewing guidelines, since these sections were not relevant for understanding the main paper, these were not reviewed with scrutiny. However, the reviewers found strong statements in the unreviewed supplementary material involving other recent work which they believe deserve close scrutiny if they are to be published. They therefore question whether the conference format is appropriate for this work, or whether the paper should instead be presented in a format to allow for closer scrutiny for all presented results (eg journal submission). However, the reviewers did not highlight anything seriously wrong with the supplementary material, therefore this cannot be considered grounds for rejection. The AC would want to stress that NeurIPS is not providing a strong stamp of approval on supplementary material since the reviewers are not strictly responsible to review all such material. Having this paper published is not an explicit sanction of the supplementary material, and the authors are advised to remove unrelated material at the camera ready to avoid potential confusion. Specifically, the reviewers asked to remove the line on a Bayesian perspective on tempering from the abstract as it isn't covered in the main paper, and perhaps to remove the section in the supplementary material as concerns were raised about it: a) the claims given are speculation and are not backed up with experimental or theoretical justification, and b) they involve a very broad definition of what is Bayesian that includes model mismatch or correcting for approximate inference etc. and this is not reflected by the sentence in the abstract.