NeurIPS 2019
Sun Dec 8th through Sat the 14th, 2019 at Vancouver Convention Center
Reviewer 1
Quality Proofs of the theorems seem correct, to the best I can judge. In the experimental part, the results are given in average but no information is given about the variation of these results Clarity The paper is well organized and clearly written. The goal of the paper as well as the results are clearly presented. Significance The paper makes a significant contribution to the study of the identifiability of PL models, even if the identifiability cases are rather limited and therefore difficult to use in practice. Moreover no experience on real data is proposed.
Reviewer 2
The model seems quite novel but it is potentially too large. To handle realistic data inputs consisting of varied k-way partial orders, one needs potentially as many as min(#partial orders, m choose k) mixture components which is rather unrealistic for MLE. Perhaps investigate other models that do not require such combinatorial explosion in no. of mixture components? Theoretical results appear correct, but as mentioned above the authors should try to more fully characterize identifiability beyond the provided corner cases. It is nice to have algorithm with consistency guarantee for 2-PL. How does one extend the algorithm beyond just the data structures stated in theorem 2? Experiments and paper motivation can have more significance if tested on real partial orders. Such public datasets are readily available, so I was scratching my head as to why this isn’t used. The paper is well-organized and clear, although the notation regarding structure mixture can be slightly confusing. Related work coverage is good.
Reviewer 3
Originality, quality, and significance: Models that are able to address multiple types of ranking data simultaneously are important, and in that regard, I consider the first steps taken by this paper to be original. Some of the identifiability results presented here generalize known results for special cases, and the authors are honest in their comparisons with related work. This line of investigation is interesting, and likely to be built upon. Clarity: The writing is clear, and the proofs are correct as far as I can make out. Overall, I think that the paper makes a nice contribution to our understanding of applying PL models to multiple types of partial orders. The theoretical results are interesting, and serve as a nice first step to understanding these models. I vote to accept the paper.