NeurIPS 2019
Sun Dec 8th through Sat the 14th, 2019 at Vancouver Convention Center
Paper ID:5005
Title:ADDIS: an adaptive discarding algorithm for online FDR control with conservative nulls

Reviewer 1

The manuscript if very well written. The problem of "error-guarantees on a stream of decisions" is of great interest. Conservative p-values mean that the data "surprised" the researcher. In my experience, this rarely happens. It seems, however, that the price of protection from such a scenario is not high. This, in my view, is the best feature of ADDIS.

Reviewer 2

The paper is very well written. It’s a pleasure to read. Online FDR control for conservative nulls is an important problem and the method is novel. The numerical studies show superior power performance compared to existing methods when the null p-values are conservative. However there is no real-data experiments in the paper. It would be interesting to see if the new method can give better discoveries in practical problems. Minor points: Line 120: FDP_LORD++ -> \hat{FDP}_LORD++ Line 158: Do you mean \lambda*\tau <= \alpha or > \alpha?

Reviewer 3

Overall, I think this is a good paper. Originality: This paper combines the work of two previous algorithms. Yet such combination does not seem to be trivial. So I believe the work is somewhat novel. Quality: The proof looks right. One comment is that the paper claims that convex CDF function would satisfy the condition of equation (3). CDF is bounded and therefore can not be a convex function on R if there is no further clarification. And some concrete examples of this claim would be nice. Clarity: It is easy to follow and understand the motivation. Significance: The paper gives a hybrid of the previous two algorithms LORD and SAFFRON and empirically shows that the new algorithm has inherited the advantages of both algorithms. In my opinion, it is interesting. However, I could be wrong since I am not an expert in this field of literature.