NeurIPS 2019
Sun Dec 8th through Sat the 14th, 2019 at Vancouver Convention Center
Paper ID:1879
Title:PC-Fairness: A Unified Framework for Measuring Causality-based Fairness

Reviewer 1

Advantages: 1. The proposed framework subsumes most previous fairness notions. It can potentially help identify, analyze and compare new fairness notions. 2. The proposed bounding method is simple and makes sense. 3. The paper is clear and easy to follow. Weakness: 1. There are many limitations of the proposed method. The proposed method assumes that the causal graphical is given. Also, the values must be discrete. 2. It would be good to show how to use the proposed method to achieve fair policy learning without "severely damaging the performance of predictive model". 3. It would be great to discuss why the fairness bound achieved by the proposed method is tighter compared with previous methods. Minor issues: line 17 irrespective their -> irrespective of their line 240 to find -> to finding Should the the numbers in Table 3 CE #of o 4 be bold? The bound of the proposed method is tighter than previous methods.

Reviewer 2

I was surprised to see that this notion of causal effect has not been defined before. Personally, I always assumed that this notion must be known, but I am unable to find a reference. Therefore, although quite trivial, it seems that this notion of causal effect has not been discussed before in the literature. I may be wrong and advise the authors to do a thorough literature search to verify this. Apart from this, I see this notion as a specific case of counterfactual effect, restricted to a set of paths. The definition should easily follow from Ilya Shpitser's paper which the authors give as well. The linear programming approach for bounding is also known as authors acknowledge. In practice, I am unsure of the impact of this work or this notion. This notion carries all the difficulties of path-based counterfactuals and more, in terms of actually evaluating it, even with interventional data. Line 17: irrespective of etc..->etc. Line 24: total effects of interventions Line 71: we don't make assumption -> we do not make assumptions Please add the relevant references to Section 2. Also, definition 3 is not clear as "performing an intervention along a path" is not formally defined. I recommend the authors to change the abbreviation PC effect as it might mislead the reader into thinking the concept is related to the infamous PC algorithm. Authors should cite "A Potential Outcomes Calculus for Identifying Conditional Path-Specific Effects" by Malinsky et al. The response function representation argued in the lines 164-170 require the assumption that the observed variables have finite support. AFTER REBUTTAL: Thank you for your responses and explaining a real-world use case, please add this to the camera ready. I still strongly recommend changing the title as this title will create a lot of confusion in the causality community. I also want to reiterate that Def. 3 is not mathematically sound, and will be unclear for the readers who are not already familiar with path-specific counterfactual literature, since intervention on a path is not formally defined. Please elaborate on how to actually evaluate this expression mathematically.

Reviewer 3

The authors attempt to unify different definitions of counterfactual fairness frameworks by characterizing them as appropriate conditioning along specific causal pathways. The main unification is simple. However, the primary contribution appears to be a method to bound the fairness effect under the unified definition of path-specific counterfactual fairness by parameterizing the causal model appropriately and mapping the estimation to a constrained optimization. A few things were not clear in this parametrization process. For example the composition in Line 168 only becomes clear later in the examples. In terms of originality, although the method is computationally intense (requiring response variables that scale exponentially as the node degrees), the contribution and originality are useful as they attempt to bound fairness in unidentifiable situations as well. Quality - The paper is technically sound although a lot more description could have been moved and/or included in supplementary material which only includes code and data as of now. This affects clarity of the paper from time to time. The related work seems adequately cited and compared to. Significance - The method is severly limited as it only works for discrete variables and as the authors note, the domain sizes of the response variables can grow significantly and quickly. While they address this to some extent, it is not clear how generalizable this method is to continuous case at all. Nevertheless it is an important contribution to counterfactual fairness literature.