The initial scores for this paper were diverging: 3: A clear reject. 6: Marginally above the acceptance threshold. 8: Top 50% of accepted NeurIPS papers. A very good submission; a clear accept. 7: A good submission; accept. 5: Marginally below the acceptance threshold. The positive points praised by the reviewers were: - Technically sound, clever, novel, simple and effective method. - Extensive evaluation on popular benchmarks that are sound and reliable. - Very well written paper. - Nice qualitative evaluation. The negative points: - Concerns regarding missing comparisons with important baselines and choice of metrics. - Limited novelty. - Missing run-time comparisons. - Missing analysis why the results are good. - Presentation: Some sections could be re-ordered, some missing details. The authors provide a rebuttal, which addresses many of the weak points. After reading the rebuttal and in the post-rebuttal discussion, R5 increases their score from 6 to 7, R1 maintains their positive rating 7, R4 still likes the paper but lowers their score from 8 to 7 taking into account comments from the other reviewers. In view of the rebuttal, the negative reviewer upgrades their score from 3 to 4 as the rebuttal clarifies some of the issues, but still has concerns “how the method performs on real settings.” The borderline negative R3 (5) did not participate in the discussion or provided a post-rebuttal comment. The final scores of the paper are: 7, 7, 7, 5, 4. The AC has looked at the reviews, rebuttal and has read the paper. The AC is convinced by the arguments of the three positive reviews and recommends Accept.