NeurIPS 2020

Finding the Homology of Decision Boundaries with Active Learning

Meta Review

The authors presents an application of active learning to the problem of finding the homology of classifier/dataset decision boundaries. The method shows benefits empirically for learning a homology and is also paired with upper bound guarantees on the required number of labels. There was a concern raised around the correctness of the label complexity proof, which the authors have suggested a fix for in the rebuttal. This has satisfied the reviewer, however, there is an additional gap between theory and the algorithm implementation pointed out. This was not deemed a critical flaw, but please do discuss it in the final version. There was also a valid criticism raised regarding the empirical evaluation of using the homology for the purpose of model selection; the investigation would be stronger there were also comparisons to baselines that match models to datasets with simpler heuristics (e.g. matching class distributions). I view the model selection application outside the immediate scope of this paper, nonetheless, a comparison to simpler baselines would strengthen the paper.