This paper introduces metrics for quantifying the fairness of a personalized intervention (with binary treatment & outcome) that are analogous to fpr/tpr in the classification setting. They show how these metrics are identifiable under the assumptions of strong-ignorability and monotone treatment response (treatment is never harmful). Developing appropriate measures of fairness for personalised interventions is an important questions, with significant potential application and this paper is well structured and clearly written. The identification results themselves are not particularly novel as similar results have been proven in other settings, but the context is new. I have read the author response and other reviewers comments and maintain that this paper is worth accepting because: ML systems recommending personalised interventions in sensitive settings, such as improving educational results or reducing recidivism are widely encountered in practise in areas where we are concerned with fairness. So this paper represents an important attempt to extend existing fairness metrics to this more complex setting. While I think it the appropriateness of quantifying fairness in terms of metrics that are not identifiable without monotonicity assumptions (even with unlimited randomised experimental data) remains up for debate, this is a debate that is worth having within the NeurIPS community. This paper is well written and argued and represents a good starting point for that debate.

While the basic ideas explored here are interesting, I have several concerns with this work: (1) The primary motivation or contributions of the paper are not well articulated. The title and writing at various places suggest "personalized interventions". But, the methods for auditing policies for fairness as well as the evaluation results are centered around scenarios where interventions are conducted in a randomized and controlled (i.e., non-personalized) way. So in what way is the work related to personalized interventions, beyond perhaps, making a case for personalized interventions to achieve fairness? (2) The focus of much of the main (8-page) paper is on unfairness measures (disparity in TPR and TNR) and how to estimate them, but not on mechanisms to achieve them. This makes me wonder how well suited the paper is to NeurIPS and if this paper would not be a better fit for a conference focussed more on fairness. Because one might argue that, while the ideas are conceptually interesting, technically the paper does not contribute much. (3) I think there is an error on line 168 -- it is hard to distinguish between p_{10} and p_{00} rather than p_{10} and p_{11}. Can you check this again?

The paper considers an important and critical societal matter and is well-written. Personalized interventions are ideal in cases where everyone has access to the same resources, but resources are limited and intervention on one’s treatment may impact interventions on other individuals in the population. In other words, decisions may interfere with each other. This has been studied in this recent paper: https://icml.cc/Conferences/2019/ScheduleMultitrack?event=4860. Also, fair personalized intervention has been considered in this paper: https://icml.cc/Conferences/2019/ScheduleMultitrack?event=5183 where the setting is very similar to the one being considered in this paper (offline optimal policy learning under fairness constraints). I was wondering what the authors’ thoughts are on these two papers? and how limited resources impact their disparate impact metrics?

Update: I have read the other reviewers' comments and the author feedback, which has adequately addressed my concerns. My overall score remains unchanged. ============ Originality. Proposition 2 seems to be the main novel observation of this work. Although it is straightforward, it is nice to point out that the monotone treatment response assumption, which is reasonable for 'positive'/beneficial interventions, allows for identification in this case. The insight seems to be present in previous work [32], so extending it to give expressions for TPR/FPR is not particularly original though still useful. The partial identification results are quite novel, though I was not sure of the practicality. How should uncertainty sets for p_10 be chosen? Is there any way to estimate how much the monotone treatment response assumption is violated in practice? Given the monotone treatment response assumption, many "non-causal" results from previous work can be applied without modification, e.g. [34]. Robust ROC and xROC extends the standard tools to the partial identification case, but it's not clear how useful these notions are/if they provide any additional insight, even in the case study provided. They seem to be too pessimistic and it's hard to interpret in the curves in Figure 2. Quality. Paper was well written. Clarity. Writing and results were clear. Figure 2 had too many curves and it was unclear how to interpret. Significance. It is important to address identifiability issues for 'fairness metrics' in applications involving interventions (and therefore missing data). Empirical verification was somewhat limited, since only 1 dataset was used.