Following the authors' response and the discussion all reviewers agree that the paper should be accepted at NeurIPS. The paper proposes a nice formulation for human-computer collaboration. The proposed technique is clearly explained and shows good performance on both a simulated toy domain and a human user study. While there are some clear limitations (experiments are performed on simplistic domains, assumes homogeneous noise and high SNR, etc.), the paper is definitely a good step forward and is likely to inspire followup work.