NeurIPS 2020

Mitigating Manipulation in Peer Review via Randomized Reviewer Assignments

Meta Review

The reviewers were initially fairly positive about the contribution of this paper. Some of the highlights are that the proposed method seems simple and practical (and, e.g., could be used right away for conference matching). During the discussion, two issues were raised: 1) The lack of formalization of the objective (this is detailed in Reviewer #5's review and post-response comments). The authors provided a very good response in their rebuttal and while some of these elements could be added to the paper, there was a consensus that a deeper investigation is warranted. 2) While this paper is interesting, it is somewhat less relevant to the core interests of NeurIPS (although it falls within the topics outlined in the CFP) and may receive more interest from the participants at other venues (e.g., IJCAI and AAAI). Also, I wanted to thank you for the AC-note you sent with your response. I agree that while both real-world evaluations and surveying the participants of a conference could add a lot of value to this work, both constitute major endeavors that are better left for future projects. Our discussion did not center around these aspects and so I believe that the reviewers agreed with your response.