NeurIPS 2019
Sun Dec 8th through Sat the 14th, 2019 at Vancouver Convention Center
Paper ID:1926
Title:Conformalized Quantile Regression

Reviewer 1

The paper is well written and the topic is highly relevant is this era of reproducibility crisis. However, its significance remains unclear wrt to the actual literature. 1- The classical conformal prediction method (section 3) is already valid for *any* regression algorithm (including quantile regression) and *any* conformity score function (including E_i in Eq. 6) (under mild assumptions e.g. exchangeability of the data and permutation invariance of the score). In that sense, the novelties of this paper rather limited and seems to be a direct application of classical conformal prediction with a quantile regression algorithm. If it is not the case, it should be interesting that the authors precisely clarify the fundamental differences between these approaches (More, the proofs seems to be exactly the same). Relatedly, the sentence in line 61-63 seems confusing. As far as I understand, standard method is not only restricted to conditional mean regression. Same remark for line 58-60, since any estimator can be used, the coverage guarantee hold regardless of the choice or accuracy (which one: accuracy in optimization or in prediction?) of the regression algorithm (also under mild assumptions). 2- To me, the discussions on the limitation in length of C(X_n+1) in Eq. 5 is also confusing. a) When one consider *full* conformal prediction set (not necessarily an interval) [mu(X_n+1) - Q_{1-alpha}(R), mu(X_n+1) + Q_{1-alpha}(R)] for all possibilities of y_n+1, the quantiles Q_{1-alpha}(R) depends on X_n+1 and its length is *not* fixed (in cases where it is an interval). Hence this limitation seems to come from the splitting strategy itself. b) The proposed conformal set C(X_n+1) in Eq. 7, has a length that is lower bounded by 2 Q_{1-alpha}(E, I_2) independently of X_n+1. Why the critics above does not hold in this case? Perhaps the locally adaptive approach can alleviate the issue. If the above points hold, the numerical experiments rather highlight the critical importance of the choice of the estimator and the conformity score used, to obtain small confidence set.

Reviewer 2

The paper is novel, well-written and important. I do not have any complaint but a few minor suggestions. (1) There are some works on linear quantile regression, which provided the non-conformalized version of the method in this paper, e.g. Zhou and Portnoy (1996, 1998). These may be worth mentioning. (2) The acronym CQR has been used for composite quantile regression (Zou and Yuan), which is another influential paper. It may be worth changing the acronym as this paper is likely to become influential too! (3) I like the point that conformalized quantile regression is better than locally adaptive conformal prediction. It may be better to design numerical experiments with highly heavy-tailed distribution (say Cauchy) to further illustrate this point. References Zhou, K. Q., & Portnoy, S. L. (1996). Direct use of regression quantiles to construct confidence sets in linear models. The Annals of Statistics, 24(1), 287-306. Zhou, K. Q., & Portnoy, S. L. (1998). Statistical inference on heteroscedastic models based on regression quantiles. Journal of Nonparametric Statistics, 9(3), 239-260. Zou, H., & Yuan, M. (2008). Composite quantile regression and the oracle model selection theory. The Annals of Statistics, 36(3), 1108-1126.

Reviewer 3

The paper provides an interesting link between Quantile Regression and Conformal Prediction that would allow to combine the strengths of both when dealing with heteroscedastic data. Both methods provide different ways of constructing prediction intervals. Quantile Regression portion of the method can use any QR method including Linear Quantile Regression, Quantile Random Forests, etc depending on the problem at hand. The experiment section demonstrated the tighter prediction intervals obtained by CQR Random Forests and CQR Neural Net.