{"title": "Equality of Opportunity in Supervised Learning", "book": "Advances in Neural Information Processing Systems", "page_first": 3315, "page_last": 3323, "abstract": "We propose a criterion for discrimination against a specified sensitive attribute in supervised learning, where the goal is to predict some target based on available features. Assuming data about the predictor, target, and membership in the protected group are available, we show how to optimally adjust any learned predictor so as to remove discrimination according to our definition. Our framework also improves incentives by shifting the cost of poor classification from disadvantaged groups to the decision maker, who can respond by improving the classification accuracy.", "full_text": "Equality of Opportunity in Supervised Learning\n\nMoritz Hardt\n\nGoogle\n\nm@mrtz.org\n\nEric Price\u2217\nUT Austin\n\necprice@cs.utexas.edu\n\nNathan Srebro\nTTI-Chicago\n\nnati@ttic.edu\n\nAbstract\n\nWe propose a criterion for discrimination against a speci\ufb01ed sensitive attribute in\nsupervised learning, where the goal is to predict some target based on available fea-\ntures. Assuming data about the predictor, target, and membership in the protected\ngroup are available, we show how to optimally adjust any learned predictor so as to\nremove discrimination according to our de\ufb01nition. Our framework also improves\nincentives by shifting the cost of poor classi\ufb01cation from disadvantaged groups to\nthe decision maker, who can respond by improving the classi\ufb01cation accuracy.\nWe enourage readers to consult the more complete manuscript on the arXiv.\n\n1\n\nIntroduction\n\nAs machine learning increasingly affects decisions in domains protected by anti-discrimination law,\nthere is much interest in algorithmically measuring and ensuring fairness in machine learning. In\ndomains such as advertising, credit, employment, education, and criminal justice, machine learning\ncould help obtain more accurate predictions, but its effect on existing biases is not well understood.\nAlthough reliance on data and quantitative measures can help quantify and eliminate existing biases,\nsome scholars caution that algorithms can also introduce new biases or perpetuate existing ones [1].\nIn May 2014, the Obama Administration\u2019s Big Data Working Group released a report [2] arguing\nthat discrimination can sometimes \u201cbe the inadvertent outcome of the way big data technologies\nare structured and used\u201d and pointed toward \u201cthe potential of encoding discrimination in automated\ndecisions\u201d. A subsequent White House report [3] calls for \u201cequal opportunity by design\u201d as a guiding\nprinciple in domains such as credit scoring.\nDespite the demand, a vetted methodology for avoiding discrimination against protected attributes\nin machine learning is lacking. A na\u00efve approach might require that the algorithm should ignore all\nprotected attributes such as race, color, religion, gender, disability, or family status. However, this\nidea of \u201cfairness through unawareness\u201d is ineffective due to the existence of redundant encodings,\nways of predicting protected attributes from other features [4].\nAnother common conception of non-discrimination is demographic parity [e.g. 5, 6, 7]. Demographic\nparity requires that a decision\u2014such as accepting or denying a loan application\u2014be independent of\nthe protected attribute. Through its various equivalent formalizations this idea appears in numerous\npapers. Unfortunately, the notion is seriously \ufb02awed on two counts [8]. First, it doesn\u2019t ensure\nfairness. The notion permits that we accept the quali\ufb01ed applicants in one demographic, but random\nindividuals in another, so long as the percentages of acceptance match. This behavior can arise\nnaturally, when there is little or no training data available for one of the demographics. Second,\ndemographic parity often cripples the utility that we might hope to achieve, especially in the common\nscenario in which an outcome to be predicated, e.g. whether the loan be will defaulted, is correlated\nwith the protected attribute. Demographic parity would not allow the ideal prediction, namely giving\nloans exactly to those who won\u2019t default. As a result, the loss in utility of introducing demographic\nparity can be substantial.\n\n\u2217Work partially performed while at OpenAI.\n\n30th Conference on Neural Information Processing Systems (NIPS 2016), Barcelona, Spain.\n\n\fIn this paper, we consider non-discrimination from the perspective of supervised learning, where the\ngoal is to predict a true outcome Y from features X based on labeled training data, while ensuring\nthe prediction is \u201cnon-discriminatory\u201d with respect to a speci\ufb01ed protected attribute A. The main\nquestion here, for which we suggest an answer, is what does it mean for such a prediction to be\nnon-discriminatory. As in the usual supervised learning setting, we assume that we have access to\nlabeled training data, in our case indicating also the protected attribute A. That is, to samples from\nthe joint distribution of (X, A, Y ). This data is used to construct a (possibly randomized) predictor\n\u02c6Y (X) or \u02c6Y (X, A), and we also use such labeled data to test for non-discriminatory.\nThe notion we propose is \u201coblivious\u201d, in that it is based only on the joint distribution, or joint statistics,\nof the true target Y , the predictions \u02c6Y , and the protected attribute A. In particular, it does not evaluate\nthe features in X nor the functional form of the predictor \u02c6Y (X) nor how it was derived. This matches\nother tests recently proposed and conducted, including demographic parity and different analyses of\ncommon risk scores. In many cases, only oblivious analysis is possible as the functional form of the\nscore and underlying training data are not public. The only information about the score is the score\nitself, which can then be correlated with the target and protected attribute. Furthermore, even if the\nfeatures or the functional form are available, going beyond oblivious analysis essentially requires\nsubjective interpretation or casual assumptions about speci\ufb01c features, which we aim to avoid.\nIn a recent concurrent work, Kleinberg, Mullainathan and Raghavan [9] showed that the only way\nfor a meaningful score that is calibrated within each group to satisfy a criterion equivalent to\nequalized odds is for the score to be a perfectly accurate predictor. This result highlights a contrast\nbetween equalized odds and other desirable properties of a score, as well the relationship between\nnondiscrimination and accuracy, which we also discuss.\n\nContributions We propose a simple,\ninterpretable, and easily checkable notion of non-\ndiscrimination with respect to a speci\ufb01ed protected attributes. We argue that, unlike demographic\nparity, our notion provides a meaningful measure of discrimination, while allowing for higher utility.\n\nUnlike demographic parity, our notion always allows for the perfectly accurate solution of (cid:98)Y = Y.\nMore broadly, our criterion is easier to achieve the more accurate the predictor (cid:98)Y is, aligning fairness\n\nwith the central goal in supervised learning of building more accurate predictors. Our notion is\nactionable, in that we give a simple and effective framework for constructing classi\ufb01ers satisfying our\ncriterion from an arbitrary learned predictor.\nOur notion can also be viewed as shifting the burden of uncertainty in classi\ufb01cation from the protected\nclass to the decision maker. In doing so, our notion helps to incentivize the collection of better features,\nthat depend more directly on the target rather then the protected attribute, and of data that allows\nbetter prediction for all protected classes.\nIn an updated and expanded paper, arXiv:1610.02413, we also capture the inherent limitations of\nour approach, as well as any other oblivious approach, through a non-identi\ufb01ability result showing that\ndifferent dependency structures with possibly different intuitive notions of fairness cannot be separated\nbased on any oblivious notion or test. We strongly encourage readers to consult arXiv:1610.02413\ninstead of this shortened presentation.\n\n2 Equalized odds and equal opportunity\n\nWe now formally introduce our \ufb01rst criterion.\n\nvariable Y. This encourages the use of features that relate to Y directly, not through A.\nAs stated, equalized odds applies to targets and protected attributes taking values in any space,\nincluding discrete and continuous spaces. But in much of our presentation we focus on binary targets\n\nDe\ufb01nition 2.1 (Equalized odds). We say that a predictor(cid:98)Y satis\ufb01es equalized odds with respect to\nprotected attribute A and outcome Y, if (cid:98)Y and A are independent conditional on Y.\nUnlike demographic parity, equalized odds allows (cid:98)Y to depend on A but only through the target\nY,(cid:98)Y and protected attributes A, in which case equalized odds is equivalent to:\nPr(cid:110)(cid:98)Y = 1 | A = 0, Y = y(cid:111) = Pr(cid:110)(cid:98)Y = 1 | A = 1, Y = y(cid:111) ,\n\ny \u2208 {0, 1}\n\n(2.1)\n\n2\n\n\fFor the outcome y = 1, the constraint requires that (cid:98)Y has equal true positive rates across the two\n\ndemographics A = 0 and A = 1. For y = 0, the constraint equalizes false positive rates. Equalized\nodds thus enforces both equal bias and equal accuracy in all demographics, punishing models that\nperform well only on the majority.\n\nEqual opportunity In the binary case, we often think of the outcome Y = 1 as the \u201cadvantaged\u201d\noutcome, such as \u201cnot defaulting on a loan\u201d, \u201cadmission to a college\u201d or \u201creceiving a promotion\u201d. A\npossible relaxation of equalized odds is to require non-discrimination only within the \u201cadvantaged\u201d\noutcome group. That is, to require that people who pay back their loan, have an equal opportunity of\ngetting the loan in the \ufb01rst place (without specifying any requirement for those that will ultimately\ndefault). This leads to a relaxation of our notion that we call \u201cequal opportunity\u201d.\n\nDe\ufb01nition 2.2 (Equal opportunity). We say that a binary predictor(cid:98)Y satis\ufb01es equal opportunity with\nrespect to A and Y if Pr{(cid:98)Y = 1 | A = 0, Y = 1} = Pr{(cid:98)Y = 1 | A = 1, Y = 1} .\n\nEqual opportunity is a weaker, though still interesting, notion of non-discrimination, and can thus\nallows for better utility.\n\nReal-valued scores Even if the target is binary, a real-valued predictive score R = f (X, A) is\noften used (e.g. FICO scores for predicting loan default), with the interpretation that higher values\n\nt. Varying this threshold changes the trade-off between sensitivity and speci\ufb01city.\nOur de\ufb01nition for equalized odds can be applied also to score functions: a score R satis\ufb01es equalized\nodds if R is independent of A given Y . If a score obeys equalized odds, then any thresholding\n\nof R correspond to greater likelihood of Y = 1 and thus a bias toward predicting(cid:98)Y = 1. A binary\nclassi\ufb01er(cid:98)Y can be obtained by thresholding the score, i.e. setting(cid:98)Y = I{R > t} for some threshold\n(cid:98)Y = I{R > t} of it also obeys equalized odds In Section 3, we will consider scores that might not\n\nsatisfy equalized odds, and see how equalized odds predictors can be derived from them by using\ndifferent (possibly randomized) thresholds depending on the value of A.\n\nOblivious measures Our notions of non-discrimination are oblivious in the following formal sense:\n\nAs a consequence of being oblivious, all the information we need to verify our de\ufb01nitions is contained\n\nparameters that can be estimated to very high accuracy from samples. We will therefore ignore the\n\n3 Achieving non-discrimination\n\nDe\ufb01nition 2.3. A property of a predictor (cid:98)Y or score R is said to be oblivious if it only depends on\nthe joint distribution of (Y, A,(cid:98)Y ) or (Y, A, R), respectively.\nin the joint distribution of predictor, protected group and outcome, ((cid:98)Y , A, Y ). In the binary case,\nwhen A and Y are reasonably well balanced, the joint distribution of ((cid:98)Y , A, Y ) is determined by 8\neffect of \ufb01nite sampling and instead assume that we know the joint distribution of ((cid:98)Y , A, Y ).\nWe now explain how to obtain an equalized odds or equal opportunity predictor (cid:101)Y from a, possibly\ndiscriminatory, learned binary predictor (cid:98)Y or score R. We envision that (cid:98)Y or R are whatever comes\nstep. Instead, we will construct a non-discriminating predictor which is derived from (cid:98)Y or R:\nDe\ufb01nition 3.1 (Derived predictor). A predictor (cid:101)Y is derived from a random variable R and the\nparticular, (cid:101)Y is independent of X conditional on (R, A).\nThe de\ufb01nition asks that the value of a derived predictor(cid:101)Y should only depend on R and the protected\nattribute, though it may introduce additional randomness. But the formulation of (cid:101)Y (that is, the\ntime in order to construct the predictor (cid:101)Y , but at prediction time we only have access to values of\n\nout of the existing training pipeline for the problem at hand. Importantly, we do not require changing\nthe training process, as this might introduce additional complexity, but rather only a post-learning\n\nprotected attribute A if it is a possibly randomized function of the random variables (R, A) alone. In\n\nfunction applied to the values of R and A), depends on information about the joint distribution of\n(R, A, Y ). In other words, this joint distribution (or an empirical estimate of it) is required at training\n\n(R, A). No further data about the underlying features X, nor their distribution, is required.\n\n3\n\n\fFigure 1: Finding the optimal equalized odds predictor (left), and equal opportunity predictor (right).\n\nLoss minimization.\n\nIt is always easy to construct a trivial predictor satisfying equalized odds, by\n\n3.1 Deriving from a binary predictor\n\nmaking decisions independent of X, A and R. For example, using the constant predictor (cid:98)Y = 0 or\n(cid:98)Y = 1. The goal, of course, is to obtain a good predictor satisfying the condition. To quantify the\nnotion of \u201cgood\u201d, we consider a loss function (cid:96) : {0, 1}2 \u2192 R that takes a pair of labels and returns a\nreal number (cid:96)((cid:98)y, y) \u2208 R which indicates the loss (or cost, or undesirability) of predicting(cid:98)y when\nthe correct label is y. Our goal is then to design derived predictors (cid:101)Y that minimize the expected\nloss E(cid:96)((cid:101)Y , Y ) subject to one of our de\ufb01nitions.\nIn designing a derived predictor from binary(cid:98)Y and A we can only set four parameters: the conditional\nprobabilities pya = Pr{(cid:101)Y = 1 | (cid:98)Y = a, A = a}. These four parameters, p = (p00, p01, p10, p11),\ntogether specify the derived predictor (cid:101)Yp. To check whether (cid:101)Yp satis\ufb01es equalized odds we need to\nThe components of \u03b3a((cid:101)Y ) are the false positive rate and the true positive rate within the demographic\nA = a. Following (2.1), (cid:101)Y satis\ufb01es equalized odds iff \u03b30((cid:101)Y ) = \u03b31((cid:101)Y ). But \u03b3a((cid:101)Yp) is just a linear\nfunction of p, with coef\ufb01cients determined by the joint distribution of (Y,(cid:98)Y , A). Since the expected\nloss E(cid:96)((cid:101)Yp, Y ) is also linear in p, we have that the optimal derived predictor can be obtained as a\n\n\u03b3a((cid:101)Y ) def= (cid:16)Pr{(cid:101)Y = 1 | A = a, Y = 0}, Pr{(cid:101)Y = 1 | A = a, Y = 1}(cid:17) .\n\nsolution to the following linear program with four variables and two equality constraints:\n\nverify the two equalities speci\ufb01ed by (2.1), for both values of y. To this end, we denote\n\n(3.1)\n\nTo better understand this linear program, let us understand the range of values \u03b3a((cid:101)Yp) can take:\nClaim 3.2. {\u03b3a((cid:101)Yp) | 0 (cid:54) p (cid:54) 1} = Pa((cid:98)Y ) def= convhull(cid:110)(0, 0), \u03b3a((cid:98)Y )\u03b3a(1 \u2212(cid:98)Y ), (1, 1)(cid:111)\nThese polytopes are visualized in Figure 1. Since each \u03b3a((cid:101)Yp), for each demographic A = a, depends\non two different coordinates of p, the choice of \u03b30 \u2208 P0 and \u03b31 \u2208 P1 is independent. Requiring\n\u03b30((cid:101)Yp) = \u03b31((cid:101)Yp) then restricts us exactly to the intersection P0 \u2229 P1, and this intersection exactly\nspeci\ufb01es the range of possible tradeoffs between the false-positive-rate and true-positive-rate for\nderived predictors(cid:101)Y (see Figure 1). Solving the linear program (3.2) amounts to \ufb01nding the tradeoff\nin P0 \u2229 P1 that optimizes the expected loss.\nFor equalized opportunity, we only require the \ufb01rst components of \u03b3 agree, removing one of the\nequality constraints from the linear program. Now, any \u03b30 \u2208 P0 and \u03b31 \u2208 P1 that are on the same\nhorizontal line are feasible.\n\n3.2 Deriving from a score function\n\nA \u201cprotected attribute blind\u201d way of deriving a binary predictor from a score R would be to threshold\n\nit, i.e. using (cid:98)Y = I{R > t}. If R satis\ufb01ed equalized odds, then so will such a predictor, and the\n\n4\n\nmin\n\np\n\nE(cid:96)((cid:101)Yp, Y )\ns.t. \u03b30((cid:101)Yp) = \u03b31((cid:101)Yp)\n\u2200y,a0 (cid:54) pya (cid:54) 1\n\n(3.2)\n\n(3.3)\n(3.4)\n\n0.00.20.40.60.81.0Pr[eY=1|A,Y=0]0.00.20.40.60.81.0Pr[eY=1|A,Y=1]Forequalodds,resultliesbelowallROCcurves.Achievableregion(A=0)Achievableregion(A=1)OverlapResultforeY=bYResultforeY=1\u2212bYEqual-oddsoptimumEqualopportunity(A=0)Equalopportunity(A=1)0.00.20.40.60.81.0Pr[eY=1|A,Y=0]0.00.20.40.60.81.0Pr[eY=1|A,Y=1]Forequalopportunity,resultslieonthesamehorizontalline\foptimal threshold should be chosen to balance false and true positive rates so as to minimize the\nexpected loss. When R does not already satisfy equalized odds, we might need to use different\nthresholds for different values of A (different protected groups), i.e. \u02dcY = I{R > tA}. As we will\nsee, even this might not be suf\ufb01cient, and we might need to introduce randomness also here.\nCentral to our study is the ROC (Receiver Operator Characteristic) curve of the score, which captures\nthe false positive and true positive (equivalently, false negative) rates at different thresholds. These are\ncurves in a two dimensional plane, where the horizontal axes is the false positive rate of a predictor\nand the vertical axes is the true positive rate. As discussed in the previous section, equalized odds can\n\nbe stated as requiring the true positive and false positive rates, (Pr{(cid:98)Y = 1 | Y = 0, A = a}, Pr{(cid:98)Y =\n1 | Y = 1, A = a}), agree between different values of a of the protected attribute. That is, that for all\nvalues of the protected attribute, the conditional behavior of the predictor is at exactly the same point\nin this space. We will therefor consider the A-conditional ROC curves\n\nCa(t) def= (cid:16)Pr{(cid:98)R > t | A = a, Y = 0}, Pr{(cid:98)R > t | A = a, Y = 1}(cid:17) .\n\nSince the ROC curves exactly specify the conditional distributions R|A, Y , a score function obeys\nequalized odds if and only if the ROC curves for all values of the protected attribute agree, that is\nCa(t) = Ca(cid:48)(t) for all values of a and t. In this case, any thresholding of R yields an equalized\nodds predictor (all protected groups are at the same point on the curve, and the same point in\nfalse/true-positive plane).\nWhen the ROC curves do not agree, we might choose different thresholds ta for the different protected\ngroups. This yields different points on each A-conditional ROC curve. For the resulting predictor\nto satisfy equalized odds, these must be at the same point in the false/true-positive plane. This is\npossible only at points where all A-conditional ROC curves intersect. But the ROC curves might\nnot all intersect except at the trivial endpoints, and even if they do, their point of intersection might\nrepresent a poor tradeoff between false positive and false negatives.\nAs with the case of correcting a binary predictor, we can use randomization to \ufb01ll the span of possible\nderived predictors and allow for signi\ufb01cant intersection in the false/true-positive plane. In particular,\nfor every protected group a, consider the convex hull of the image of the conditional ROC curve:\n\n(3.5)\nThe de\ufb01nition of Da is analogous to the polytope Pa in the previous section, except that here we\ndo not consider points below the main diagonal (line from (0, 0) to (1, 1)), which are worse than\n\u201crandom guessing\u201d and hence never desirable for any reasonable loss function.\n\ndef= convhull{Ca(t) : t \u2208 [0, 1]}\n\nDa\n\nDeriving an optimal equalized odds threshold predictor. Any point in the convex hull Da rep-\nresents the false/true positive rates, conditioned on A = a, of a randomized derived predictor based\non R. In particular, since the space is only two-dimensional, such a predictor \u02dcY can always be taken\nto be a mixture of two threshold predictors (corresponding to the convex hull of two points on the\nROC curve). Conditional on A = a, the predictor \u02dcY behaves as\n\n\u02dcY = I{R > Ta} ,\n\nwhere Ta is a randomized threshold assuming the value ta with probability p\nand the value ta with\na\nprobability pa. In other words, to construct an equalized odds predictor, we should choose a point in\nthe intersection of these convex hulls, \u03b3 = (\u03b3[0], \u03b3[1]) \u2208 \u2229aDa, and then for each protected group\nrealize the true/false-positive rates \u03b3 with a (possible randomized) predictor \u02dcY |(A = a) = I{R > Ta}\nresulting in the predictor \u02dcY = Pr I{R > TA}. For each group a, we either use a \ufb01xed threshold\nTa = ta or a mixture of two thresholds ta < ta. In the latter case, if A = a and R < ta we always\nset \u02dcY = 0, if R > ta we always set \u02dcY = 1, but if ta < R < ta, we \ufb02ip a coin and set \u02dcY = 1 with\nprobability p\nThe feasible set of false/true positive rates of possible equalized odds predictors is thus the intersection\nof the areas under the A-conditional ROC curves, and above the main diagonal (see Figure 2).\nSince for any loss function the optimal false/true-positive rate will always be on the upper-left\nboundary of this feasible set, this is effectively the ROC curve of the equalized odds predictors.\nThis ROC curve is the pointwise minimum of all A-conditional ROC curves. The performance of\n\na\n\n.\n\n5\n\n\fFigure 2: Finding the optimal equalized odds threshold predictor (middle), and equal opportunity threshold\npredictor (right). For the equal opportunity predictor, within each group the cost for a given true positive rate\nis proportional to the horizontal gap between the ROC curve and the pro\ufb01t-maximizing tangent line (i.e., the\ntwo curves on the left plot), so it is a convex function of the true positive rate (right). This lets us optimize it\nef\ufb01ciently with ternary search.\nan equalized odds predictor is thus determined by the minimum performance among all protected\ngroups. Said differently, requiring equalized odds incentivizes the learner to build good predictors for\nall classes. For a given loss function, \ufb01nding the optimal tradeoff amounts to optimizing (assuming\nw.l.o.g. (cid:96)(0, 0) = (cid:96)(1, 1) = 0):\n\nmin\n\n\u2200a : \u03b3\u2208Da\n\n\u03b3[0](cid:96)(1, 0) + (1 \u2212 \u03b3[1])(cid:96)(0, 1)\n\n(3.6)\n\nThis is no longer a linear program, since Da are not polytopes, or at least are not speci\ufb01ed as such.\nNevertheless, (3.6) can be ef\ufb01ciently optimized numerically using ternary search.\nFor an optimal equation opportunity derived predictor the construction is similar, except its\nsuf\ufb01cient to \ufb01nd points in Da that are on the same horizontal line. Assuming continuity of the\nconditional ROC curves, we can always take points on the ROC curve Ca itself and no randomization\nis necessary.\n\n4 Bayes optimal predictors\n\nIn this section, we develop the theory a theory for non-discriminating Bayes optimal classi\ufb01cation.\nWe will \ufb01rst show that a Bayes optimal equalized odds predictor can be obtained as an derived\nthreshold predictor of the Bayes optimal regressor. Second, we quantify the loss of deriving an\nequalized odds predictor based on a regressor that deviates from the Bayes optimal regressor. This\ncan be used to justify the approach of \ufb01rst training classi\ufb01ers without any fairness constraint, and\nthen deriving an equalized odds predictor in a second step.\nDe\ufb01nition 4.1 (Bayes optimal regressor). Given random variables (X, A) and a target variable Y,\nthe Bayes optimal regressor is R = arg minr(x,a) E(cid:2)(Y \u2212 r(X, A))2(cid:3) = r\u2217(X, A) with r\u2217(x, a) =\nE[Y | X = x, A = a].\nThe Bayes optimal classi\ufb01er, for any proper loss, is then a threshold predictor of R, where the\nthreshold depends on the loss function (see, e.g., [10]). We will extend this result to the case where\nwe additionally ask the classi\ufb01er to satisfy an oblivious property:\nProposition 4.2. For any source distribution over (Y, X, A) with Bayes optimal regressor R(X, A),\nany loss function, and any oblivious property C, there exists a predictor Y \u2217(R, A) such that:\n\n1. Y \u2217 is an optimal predictor satisfying C. That is, E(cid:96)(Y \u2217, Y ) (cid:54) E(cid:96)((cid:98)Y , Y ) for any predictor\n(cid:98)Y (X, A) which satis\ufb01es C.\n\n2. Y \u2217 is derived from (R, A).\n\nCorollary 4.3 (Optimality characterization). An optimal equalized odds predictor can be derived\nfrom the Bayes optimal regressor R and the protected attribute A. The same is true for an optimal\nequal opportunity predictor.\n\n6\n\n0.00.20.40.60.81.0Pr[eY=1|A,Y=0]0.00.20.40.60.81.0Pr[eY=1|A,Y=1]Withineachgroup,maxpro\ufb01tisatangentoftheROCcurve0.00.20.40.60.81.0Pr[eY=1|A,Y=0]0.00.20.40.60.81.0Pr[eY=1|A,Y=1]Equaloddsmakestheaveragevectortangenttotheinterior0.00.20.40.60.81.0Costofbestsolutionforgiventruepositiverate0.00.20.40.60.81.0Pr[eY=1|A,Y=1]EqualopportunitycostisconvexfunctionofTPrateA=0A=1AverageOptimal\fWe can furthermore show that if we can approximate the (unconstrained) Bayes optimal regressor\nwell enough, then we can also construct a nearly optimal non-discriminating classi\ufb01er:\nDe\ufb01nition 4.4. We de\ufb01ne the conditional Kolmogorov distance between two random variables\nR, R(cid:48)\n\n\u2208 [0, 1] in the same probability space as A and Y as:\n\ndK(R, R(cid:48)) def= max\n\na,y\u2208{0,1} sup\n\nt\u2208[0,1]|Pr{R > t | A = a, Y = y} \u2212 Pr{R(cid:48) > t | A = a, Y = y}| .\nTheorem 4.5 (Near optimality). Assume that (cid:96) is a bounded loss function, and let (cid:98)R \u2208 [0, 1] be an\nodds predictor (cid:98)Y derived from ((cid:98)R, A) such that\n\narbitrary random variable. Then, there is an optimal equalized odds predictor Y \u2217 and an equalized\n\nwhere R\u2217 is the Bayes optimal regressor. The same claim is true for equal opportunity.\n\nE(cid:96)((cid:98)Y , Y ) (cid:54) E(cid:96)(Y \u2217, Y ) + 2\u221a2 \u00b7 dK((cid:98)R, R\u2217) ,\n\n(4.1)\n\n5 Case study: FICO scores\n\nFICO scores are a proprietary classi\ufb01er widely used in the United States to predict credit worthi-\nness [11]. These scores, ranging from 300 to 850, try to predict credit risk; they form our score\nR. People were labeled as in default if they failed to pay a debt for at least 90 days on at least one\naccount in the ensuing 18-24 month period; this gives an outcome Y . Our protected attribute A\nis race, which is restricted to four values: Asian, white non-Hispanic (labeled \u201cwhite\u201d in \ufb01gures),\nHispanic, and black. FICO scores are complicated proprietary classi\ufb01ers based on features, like\nnumber of bank accounts kept, that could interact with culture and race.\n\nFigure 3: These two marginals, and the number of people per group, constitute our input data.\n\nTo illustrate the effect of non-discrimination on utility we used a loss in which false positives (giving\nloans to people that default on any account) is 82/18 as expensive as false negatives (not giving a loan\nto people that don\u2019t default). Given the marginal distributions for each group (Figure 3), we can then\nstudy the optimal pro\ufb01t-maximizing classi\ufb01er under \ufb01ve different constraints on allowed predictors:\n\u2022 Max pro\ufb01t has no fairness constraints, and will pick for each group the threshold that\nmaximizes pro\ufb01t. This is the score at which 82% of people in that group do not default.\n\u2022 Race blind requires the threshold to be the same for each group. Hence it will pick the\nsingle threshold at which 82% of people do not default overall.\n\u2022 Demographic parity picks for each group a threshold such that the fraction of group\nmembers that qualify for loans is the same.\n\u2022 Equal opportunity picks for each group a threshold such that the fraction of non-defaulting\ngroup members that qualify for loans is the same.\n\u2022 Equalized odds requires both the fraction of non-defaulters that qualify for loans and the\nfraction of defaulters that qualify for loans to be constant across groups. This might require\nrandomizing between two thresholds for each group.\n\nOur proposed fairness de\ufb01nitions give thresholds between those of max-pro\ufb01t/race-blind thresholds\nand of demographic parity. Figure 4 shows the thresholds used by each predictor, and Figure 5 plots\nthe ROC curves for each group, and the per-group false and true positive rates for each resulting\npredictor. Differences in the ROC curve indicate differences in predictive accuracy between groups\n(not differences in default rates), demonstrating that the majority (white) group is classi\ufb01ed more\naccurately than other.\n\n7\n\n300400500600700800FICOscore020406080100Non-defaultrateNon-defaultratebyFICOscoreAsianWhiteHispanicBlack300400500600700800900FICOscore0.00.20.40.60.81.0FractionofgroupbelowCDFofFICOscorebygroupAsianWhiteHispanicBlack\fFigure 4: FICO thresholds for various de\ufb01nitions of fairness. The equal odds method does not give a single\n\nthreshold, but instead Pr[(cid:98)Y = 1 | R, A] increases over some not uniquely de\ufb01ned range; we pick the one\n\ncontaining the fewest people.\n\nFigure 5: The ROC curve for using FICO score to identify non-defaulters. Within a group, we can achieve any\nconvex combination of these outcomes. Equality of opportunity picks points along the same horizontal line.\nEqual odds picks a point below all lines.\n\nWe can compute the pro\ufb01t achieved by each method, as a fraction of the max pro\ufb01t achievable. A\nrace blind threshold gets 99.3% of the maximal pro\ufb01t, equal opportunity gets 92.8%, equalized odds\ngets 80.2%, and demographic parity only 69.8%.\n\n6 Conclusions\n\nWe proposed a fairness measure that accomplishes two important desiderata. First, it remedies the\nmain conceptual shortcomings of demographic parity as a fairness notion. Second, it is fully aligned\nwith the central goal of supervised machine learning, that is, to build higher accuracy classi\ufb01ers.\nOur notion requires access to observed outcomes such as default rates in the loan setting. This\nis precisely the same requirement that supervised learning generally has. The broad success of\nsupervised learning demonstrates that this requirement is met in many important applications. That\nsaid, having access to reliable \u201clabeled data\u201d is not always possible. Moreover, the measurement of\nthe target variable might in itself be unreliable or biased. Domain-speci\ufb01c scrutiny is required in\nde\ufb01ning and collecting a reliable target variable.\nRequiring equalized odds creates an incentive structure for the entity building the predictor that aligns\nwell with achieving fairness. Achieving better prediction with equalized odds requires collecting\nfeatures that more directly capture the target, unrelated to its correlation with the protected attribute.\nAn equalized odds predictor derived from a score depends on the pointwise minimum ROC curve\namong different protected groups, encouraging constructing of predictors that are accurate in all\ngroups, e.g., by collecting data appropriately or basing prediction on features predictive in all groups.\nAn important feature of our notion is that it can be achieved via a simple and ef\ufb01cient post-processing\nstep. In fact, this step requires only aggregate information about the data and therefore could even be\ncarried out in a privacy-preserving manner (formally, via Differential Privacy).\n\n8\n\n020406080100Within-groupFICOscorepercentileMaxpro\ufb01tSinglethresholdOpportunityDemographyEqualoddsFICOscorethresholds(within-group)300400500600700800FICOscoreMaxpro\ufb01tSinglethresholdOpportunityDemographyEqualoddsFICOscorethresholds(raw)AsianWhiteHispanicBlack0.00.20.40.60.81.0Fractiondefaultersgettingloan0.00.20.40.60.81.0Fractionnon-defaultersgettingloanPer-groupROCcurveclassifyingnon-defaultersusingFICOscoreAsianWhiteHispanicBlack0.000.050.100.150.200.25Fractiondefaultersgettingloan0.40.50.60.70.80.91.0Fractionnon-defaultersgettingloanZoomedinviewMaxpro\ufb01tSinglethresholdOpportunityEqualodds\fReferences\n[1] Solon Barocas and Andrew Selbst. Big data\u2019s disparate impact. California Law Review, 104,\n\n2016.\n\n[2] John Podesta, Penny Pritzker, Ernest J. Moniz, John Holdren, and Jefrey Zients. Big data:\n\nSeizing opportunities and preserving values. Executive Of\ufb01ce of the President, May 2014.\n\n[3] Big data: A report on algorithmic systems, opportunity, and civil rights. Executive Of\ufb01ce of the\n\nPresident, May 2016.\n\n[4] Dino Pedreshi, Salvatore Ruggieri, and Franco Turini. Discrimination-aware data mining. In\n\nProc. 14th ACM SIGKDD, 2008.\n\n[5] T. Calders, F. Kamiran, and M. Pechenizkiy. Building classi\ufb01ers with independency constraints.\n\nIn In Proc. IEEE International Conference on Data Mining Workshops, pages 13\u201318, 2009.\n\n[6] Indre Zliobaite. On the relation between accuracy and fairness in binary classi\ufb01cation. CoRR,\n\nabs/1505.05723, 2015.\n\n[7] Muhammad Bilal Zafar, Isabel Valera, Manuel Gomez Rodriguez, and Krishna P Gummadi.\n\nLearning fair classi\ufb01ers. CoRR, abs:1507.05259, 2015.\n\n[8] Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Richard S. Zemel. Fairness\n\nthrough awareness. In Proc. ACM ITCS, pages 214\u2013226, 2012.\n\n[9] Jon M. Kleinberg, Sendhil Mullainathan, and Manish Raghavan. Inherent trade-offs in the fair\n\ndetermination of risk scores. CoRR, abs/1609.05807, 2016.\n\n[10] Larry Wasserman. All of Statistics: A Concise Course in Statistical Inference. Springer, 2010.\n\n[11] US Federal Reserve. Report to the congress on credit scoring and its effects on the availability\n\nand affordability of credit, 2007.\n\n9\n\n\f", "award": [], "sourceid": 1654, "authors": [{"given_name": "Moritz", "family_name": "Hardt", "institution": "Google Brain"}, {"given_name": "Eric", "family_name": "Price", "institution": "The University of Texas at Austin"}, {"given_name": "Eric", "family_name": "Price", "institution": null}, {"given_name": "Nati", "family_name": "Srebro", "institution": "TTI-Chicago"}]}