NeurIPS 2020

Bayes Consistency vs. H-Consistency: The Interplay between Surrogate Loss Functions and the Scoring Function Class

Meta Review

Following the author response, there was broad consensus on the strengths of this submission: + studies a fundamental theoretical problem: while the issue of H realisable consistency is somewhat niche, furthering the understanding of the properties of surrogate losses is of clear interest, and progress in this space is welcome + the proposed fix to the inconsistency of common multiclass surrogates is intuitive, precisely studied theoretically, and demonstrated to be valuable empirically + the paper is very well-written and explained The discussion also revolved around some concerns, particularly raised by one reviewer: - mixed empirical results: in particular, the proposed fix does not consistently improve upon the original function class - potential limited scope of the theoretical results - potential redundancies with definitions and presentation The first point above was agreed on by other reviewers. However, their scores were unchanged as it was felt the results were sufficiently to furnish the theoretical results, which is the key point of the paper (as noted by the authors in Line 284). On my reading, I agree that the empirical results are sufficient for a primarily theoretical work. While there was no universal consensus on the second point above, on my reading I agree with the majority view of the other reviewers: the work appears a sufficiently self-contained contribution to an interesting problem, and the simplicity of the final solution is a salient feature. The resolution to the counter-intuitive findings of the original H realisable consistency paper are an important fundamental contribution. The third point above is minor, and somewhat at odds with the general positive scores other reviewers provided to the paper's readability. The authors are nonetheless encouraged to re-evaluate scope of trimming down some of the definitions. In summary, my recommendation is for this paper to be accepted.