NeurIPS 2020

Off-Policy Evaluation and Learning for External Validity under a Covariate Shift

Meta Review

The reviewers found the paper "interesting," "significant," "important," and "elegant," and I wholeheartedly agree with their assessment. For instance, Reviewer 1 says: "The thoroughness of results and clarity of exposition is impressive." Indeed, the masterfulness in applying semiparametric efficiency theory and the construction of doubly robust estimators are quite impressive. Even though some of the logic in these derivations is shared across the shifted and non-shifted (standard) domains, as noted by Reviewer 2, the application to this setting is quite compelling, and the results should be useful in practice. On the other hand, Reviewer 3 was a bit more critical regarding novelty but agrees that these results are applicable in "realistic and important" scenarios. Further, Reviewer 4 is positive about the paper while noting that the work should be put into context. In particular, the reviewer highlights that this is one specific instance of a transportability estimator (Bareinboim and Pearl, PNAS, 2016) (BP. henceforth). Even though there is a generic discussion of possible connections with the related literature, readers would benefit from a more detailed discussion and better articulation of what the paper is doing with respect to this broader context. In fact, this particular estimator holds when the source of variations regarding the covariates changes across settings while the mechanisms underlying the treatment and outcome variables remain invariant. In other words, the treatment A is S-ignorable w.r.t to outcome Y given the set of covariates A. The assumption of S-ignorability implies BP's Eq. 15, which seems to be equivalent to the re-weighting discussed in the paper (but for the relabeling of the variables). Transportability theory, on the other hand, delineates the conditions under which any type of extrapolation could be performed while guaranteeing consistency, and S-ignorability is just one among the viable identification strategies. The work in the paper is a significant first step. It leads to the interesting question of whether the new machinery developed could be extended to other transportability functionals to construct the corresponding double robust and efficient estimators. After all, consider reading the reviews carefully and consider their suggestions in the final version of the manuscript. All in all, this is an excellent piece of work, and my recommendation is "strong accept."