{"title": "Average Individual Fairness: Algorithms, Generalization and Experiments", "book": "Advances in Neural Information Processing Systems", "page_first": 8242, "page_last": 8251, "abstract": "We propose a new family of fairness definitions for classification problems that combine some of the best properties of both statistical and individual notions of fairness. We posit not only a distribution over individuals, but also a distribution over (or collection of) classification tasks. We then ask that standard statistics (such as error or false positive/negative rates) be (approximately) equalized across individuals, where the rate is defined as an expectation over the classification tasks. Because we are no longer averaging over coarse groups (such as race or gender), this is a semantically meaningful individual-level constraint. Given a sample of individuals and problems, we design an oracle-efficient algorithm (i.e. one that is given access to any standard, fairness-free learning heuristic) for the fair empirical risk minimization task. We also show that given sufficiently many samples, the ERM solution generalizes in two directions: both to new individuals, and to new classification tasks, drawn from their corresponding distributions. Finally we implement our algorithm and empirically verify its effectiveness.", "full_text": "Average Individual Fairness:\n\nAlgorithms, Generalization and Experiments\n\nMichael Kearns\n\nUniversity of Pennsylvania\nmkearns@cis.upenn.edu\n\nAaron Roth\n\nUniversity of Pennsylvania\naaroth@cis.upenn.edu\n\nSaeed Shari\ufb01-Malvajerdi\nUniversity of Pennsylvania\n\nsaeedsh@wharton.upenn.edu\n\nAbstract\n\nWe propose a new family of fairness de\ufb01nitions for classi\ufb01cation problems that\ncombine some of the best properties of both statistical and individual notions of\nfairness. We posit not only a distribution over individuals, but also a distribution\nover (or collection of) classi\ufb01cation tasks. We then ask that standard statistics\n(such as error or false positive/negative rates) be (approximately) equalized across\nindividuals, where the rate is de\ufb01ned as an expectation over the classi\ufb01cation tasks.\nBecause we are no longer averaging over coarse groups (such as race or gender),\nthis is a semantically meaningful individual-level constraint. Given a sample of\nindividuals and problems, we design an oracle-ef\ufb01cient algorithm (i.e. one that is\ngiven access to any standard, fairness-free learning heuristic) for the fair empirical\nrisk minimization task. We also show that given suf\ufb01ciently many samples, the\nERM solution generalizes in two directions: both to new individuals, and to new\nclassi\ufb01cation tasks, drawn from their corresponding distributions. Finally we\nimplement our algorithm and empirically verify its effectiveness.\n\n1\n\nIntroduction\n\nThe community studying fairness in machine learning has yet to settle on de\ufb01nitions. At a high level,\nexisting de\ufb01nitional proposals can be divided into two groups: statistical fairness de\ufb01nitions and\nindividual fairness de\ufb01nitions. Statistical fairness de\ufb01nitions partition individuals into \u201cprotected\ngroups\u201d (often based on race, gender, or some other binary protected attribute) and ask that some\nstatistic of a classi\ufb01er (error rate, false positive rate, positive classi\ufb01cation rate, etc.) be approximately\nequalized across those groups. In contrast, individual de\ufb01nitions of fairness have no notion of\n\u201cprotected groups\u201d, and instead ask for constraints that bind on pairs of individuals. These constraints\ncan have the semantics that \u201csimilar individuals should be treated similarly\u201d (Dwork et al. (2012)), or\nthat \u201cless quali\ufb01ed individuals should not be preferentially favored over more quali\ufb01ed individuals\u201d\n(Joseph et al. (2016)). Both families of de\ufb01nitions have serious problems, which we will elaborate on.\nBut in summary, statistical de\ufb01nitions of fairness provide only very weak promises to individuals,\nand so do not have very strong semantics. Existing proposals for individual fairness guarantees, on\nthe other hand, have very strong semantics, but have major obstacles to deployment, requiring strong\nassumptions on either the data generating process or on society\u2019s ability to instantiate an agreed-upon\nfairness metric.\nStatistical de\ufb01nitions of fairness are the most popular in the literature, in large part because they can\nbe easily checked and enforced on arbitrary data distributions. For example, a popular de\ufb01nition\n(Hardt et al. (2016); Kleinberg et al. (2017); Chouldechova (2017)) asks that a classi\ufb01er\u2019s false\npositive rate should be equalized across the protected groups. This can sound attractive: in settings\nin which a positive classi\ufb01cation leads to a bad outcome (e.g. incarceration), it is the false positives\nthat are harmed by the errors of the classi\ufb01er, and asking that the false positive rate be equalized\nacross groups is asking that the harm caused by the algorithm should be proportionately spread\nacross protected populations. But the meaning of this guarantee to an individual is limited, because\n\n33rd Conference on Neural Information Processing Systems (NeurIPS 2019), Vancouver, Canada.\n\n\fthe word rate refers to an average over the population. To see why this limits the meaning of the\nguarantee, consider the example given in Kearns et al. (2018): imagine a society that is equally split\nbetween gender (Male, Female) and race (Blue, Green). Under the constraint that false positive\nrates be equalized across both race and gender, a classi\ufb01er may incarcerate 100% of blue men and\ngreen women, and 0% of green men and blue women. This equalizes the false positive rate across all\nprotected groups, but is cold comfort to any individual blue man and green woman. This effect isn\u2019t\nmerely hypothetical \u2014 Kearns et al. (2018, 2019) showed similar effects when using off-the-shelf\nfairness constrained learning techniques on real datasets.\nIndividual de\ufb01nitions of fairness, on the other hand, can have strong individual level semantics. For\nexample, the constraint imposed by Joseph et al. (2016, 2018) in online classi\ufb01cation problems\nimplies that the false positive rate must be equalized across all pairs of individuals who (truly) have\nnegative labels. Here the word rate has been rede\ufb01ned to refer to an expectation over the randomness\nof the classi\ufb01er, and there is no notion of protected groups. This kind of constraint provides a strong\nindividual level promise that one\u2019s risk of being harmed by the errors of the classi\ufb01er are no higher\nthan they are for anyone else. Unfortunately, in order to non-trivially satisfy a constraint like this, it\nis necessary to make strong realizability assumptions.\n\n1.1 Our results\n\nWe propose an alternative de\ufb01nition of individual fairness that avoids the need to make assumptions\non the data generating process, while giving the learning algorithm more \ufb02exibility to satisfy it in\nnon-trivial ways. We consider that in many applications each individual will be subject to decisions\nmade by many classi\ufb01cation tasks over a given period of time, not just one. For example, internet\nusers are shown a large number of targeted ads over the course of their usage of a platform, not\njust one: the properties of the advertisers operating in the platform over a period of time are not\nknown up front, but have some statistical regularities. Public school admissions in cities like New\nYork are handled by a centralized match: students apply not just to one school, but to many, who\ncan each make their own admissions decisions (Abdulkadiro\u02d8glu et al. (2005)). We model this by\nimagining that not only is there an unknown distribution P over individuals, but there is an unknown\ndistribution Q over classi\ufb01cation problems (each of which is represented by an unknown mapping\nfrom individual features to target labels). With this model in hand, we can now ask that the error rates\n(or false positive or negative rates) be equalized across all individuals \u2014 where now rate is de\ufb01ned as\nthe average over classi\ufb01cation tasks drawn from Q of the probability of a particular individual being\nincorrectly classi\ufb01ed.\nWe then derive a new oracle-ef\ufb01cient algorithm for satisfying this guarantee in-sample, and prove\nnovel generalization guarantees showing that the guarantees of our algorithm hold also out of sample.\nOracle ef\ufb01ciency is an attractive framework in which to circumvent the worst-case hardness of even\nunconstrained learning problems, and focus on the additional computational dif\ufb01culty imposed by\nfairness constraints. It assumes the existence of \u201coracles\u201d that can solve weighted classi\ufb01cation\nproblems absent fairness constraints, and asks for ef\ufb01cient reductions from the fairness constrained\nlearning problems to unconstrained problems. This has become a popular technique in the fair\nmachine learning literature (see e.g. Agarwal et al. (2018); Kearns et al. (2018)) \u2014 and one that\noften leads to practical algorithms. The generalization guarantees we prove require the development\nof new techniques because they refer to generalization in two orthogonal directions \u2014 over both\nindividuals and classi\ufb01cation problems. Our algorithm is run on a sample of n individuals sampled\nfrom P and m problems sampled from Q. It is given access to an oracle (in practice, implemented\nwith a heuristic) for solving ordinary cost sensitive classi\ufb01cation problems over some hypothesis\nspace H. The algorithm runs in polynomial time (it performs only elementary calculations except for\ncalls to the learning oracle, and makes only a polynomial number of calls to the oracle) and returns\na mapping from problems to hypotheses that have the following properties, so long as n and m are\nsuf\ufb01ciently large (polynomial in the VC-dimension of H and the desired error parameters): For any\n\u03b1, with high probability over the draw of the n individuals from P and the m problems from Q\n\n1. Accuracy: the error rate (computed in expectation over new individuals x \u223c P and new\nproblems f \u223c Q) is within O(\u03b1) of the optimal mapping from problems to classi\ufb01ers in H,\nsubject to the constraint that for every pair of individuals x, x(cid:48) in the support of P, the error\nrates (or false positive or negative rates) (computed in expectation over problems f \u223c Q) on\nx and x(cid:48) differ by at most \u03b1.\n\n2\n\n\findividual\nproblem\n\nspace\nX\nF\n\nelement\nx \u2208 X\nf \u2208 F\n\nP\nQ\n\ndata set\nX = {xi}n\nF = {fj}m\n\ni=1\n\nj=1\n\nTable 1: Summary of notations for individuals vs. problems\nsample size\n\ndistribution\n\nempirical dist.\n\n(cid:98)P = U (X)\n(cid:98)Q = U (F )\n\nn\n\nm\n\n2. Fairness: with probability 1 \u2212 \u03b2 over the draw of new individuals x, x(cid:48) \u223c P, the error rate\n(or false positive or negatives rates) of the output mapping (computed in expectation over\nproblems f \u223c Q) on x will be within O(\u03b1) of that of x(cid:48).\n\nThe mapping from new classi\ufb01cation problems to hypotheses that we \ufb01nd is derived from the dual\nvariables of the linear program representing our empirical risk minimization task, and we crucially\nrely on the structure of this mapping to prove our generalization guarantees for new problems f \u223c Q.\n\n1.2 Additional related work\n\nThe literature on fairness in machine learning has become much too large to comprehensively\nsummarize, but see Mitchell et al. (2018) for a recent survey. Here we focus on the most conceptually\nrelated work, which has aimed to bridge the gap between the immediate applicability of statistical\nde\ufb01nitions of fairness with the strong individual level semantics of individual notions of fairness.\nOne strand of this literature focuses on the \u201cmetric fairness\u201d de\ufb01nition \ufb01rst proposed by Dwork et al.\n(2012), and aims to ease the assumption that the learning algorithm has access to a task speci\ufb01c\nfairness metric. Kim et al. (2018a) imagine access to an oracle which can provide unbiased estimates\nto the metric distance between any pair of individuals, and show how to use this to satisfy a statistical\nnotion of fairness representing \u201caverage metric fairness\u201d over pre-de\ufb01ned groups. Gillen et al. (2018)\nstudy a contextual bandit learning setting in which a human judge points out metric fairness violations\nwhenever they occur, and show that with this kind of feedback (under assumptions about consistency\nwith a family of metrics), it is possible to quickly converge to the optimal fair policy. Yona and\nRothblum (2018) consider a PAC-based relaxation of metric fair learning, and show that empirical\nmetric-fairness generalizes to out-of-sample metric fairness. Another strand of this literature has\nfocused on mitigating the problems that arise when statistical notions of fairness are imposed over\ncoarsely de\ufb01ned groups, by instead asking for statistical notions of fairness over exponentially many\nor in\ufb01nitely many groups with a well de\ufb01ned structure. This line includes H\u00e9bert-Johnson et al.\n(2018) (focusing on calibration), Kearns et al. (2018) (focusing on false positive and negative rates),\nand Kim et al. (2018b) (focusing on error rates).\n\n2 Model and preliminaries\nWe model each individual in our framework by a vector of features x \u2208 X , and we let each learning\nproblem 1 be represented by a binary function f \u2208 F mapping X to {0, 1}. We assume probability\nmeasures P and Q over X and F, respectively. In the training phase there is a \ufb01xed (across problems)\ni=1 of n individuals sampled independently from P for which we have available labels\nset X = {xi}n\nj=1 drawn independently from Q 2. Therefore, a\ncorresponding to m tasks represented by F = {fj}m\ntraining data set of n individuals X and m learning tasks F takes the form: S = {xi, (fj(xi))m\nj=1}n\ni=1.\nWe summarize the notations we use for individuals and problems in Table 1.\nIn general F will be unknown. We will aim to solve the (agnostic) learning problem over a hypothesis\nclass H, which need bear no relationship to F. We will allow for randomized classi\ufb01ers, which we\nmodel as learning over \u2206(H), the probability simplex over H. We assume throughout that H contains\nthe constant classi\ufb01ers h0 and h1 where h0(x) = 0 and h1(x) = 1 for all x. Unlike usual learning\nsettings where the primary goal is to learn a single hypothesis p \u2208 \u2206(H), our objective is to learn a\nmapping \u03c8 \u2208 \u2206(H)F that maps learning tasks f \u2208 F represented as new labellings of the training\ndata to hypotheses p \u2208 \u2206(H). We will therefore have to formally de\ufb01ne the error rates incurred\nby a mapping \u03c8 and use them to formalize a learning task subject to our proposed fairness notion.\nFor a mapping \u03c8, we write \u03c8f to denote the classi\ufb01er corresponding to f under the mapping, i.e.,\n\n1We will use the terms: problem, task, and labeling interchangeably.\n2Throughout we will use subscript i to denote individuals and j to denote learning problems.\n\n3\n\n\f\u03c8f = \u03c8 (f ) \u2208 \u2206(H). Notice in the training phase, there are only m learning problems to be solved,\nand therefore, the corresponding empirical problem reduces to learning m randomized classi\ufb01ers.\nIn general, learning m speci\ufb01c classi\ufb01ers for the training problems will not yield any generalizable\nrule mapping new problems to classi\ufb01ers \u2014 but the speci\ufb01c algorithm we propose for empirical risk\nminimization will induce such a mapping, via a dual representation of the empirical risk minimizer.\nDe\ufb01nition 2.1 (Individual and Overall Error Rates). For a mapping \u03c8 \u2208 \u2206(H)F and distributions\n\u03c8 and err (\u03c8;P,Q) = E\n\nP and Q: E (x, \u03c8;Q) = Ef\u223cQ(cid:2)Ph\u223c\u03c8f [h(x) (cid:54)= f (x)](cid:3) is the individual error rate of x incurred by\n\nx\u223cP [E (x, \u03c8;Q)] is the overall error rate of \u03c8.\n\nIn the body of this paper, we will focus on a fairness constraint that asks that the individual error\nrate should be approximately equalized across all individuals. In the supplement, we extend our\ntechniques to equalizing false positive and negative rates across individuals.\nDe\ufb01nition 2.2 (Average Individual Fairness (AIF)). We say a mapping \u03c8 \u2208 \u2206(H)F satis\ufb01es \u201c(\u03b1, \u03b2)-\nAIF\u201d (reads (\u03b1, \u03b2)-approximate Average Individual Fairness) with respect to the distributions (P,Q)\nif there exists \u03b3 \u2265 0 such that: Px\u223cP (|E (x, \u03c8;Q) \u2212 \u03b3| > \u03b1) \u2264 \u03b2.\nWe brie\ufb02y \ufb01x some notation: 1 [A] represents the indicator function of event A. For n \u2208 N,\n[n] = {1, 2, . . . , n}. U (S) represents the uniform distribution over S. For a mapping \u03c8 : A \u2192 B\nand A(cid:48) \u2286 A, \u03c8|A(cid:48) represents \u03c8 restricted to the domain A(cid:48). dH denotes the VC dimension of the\nclass H. CSC(H) denotes a cost sensitive classi\ufb01cation oracle for H:\nDe\ufb01nition 2.3 (Cost Sensitive Classi\ufb01cation (CSC) in H). Let D = {xi, c1\ni}n\ni=1 denote a\ni are the costs of classifying xi as positive (1) and\ndata set of n individuals xi where c1\nnegative (0) respectively. Given D, the cost sensitive classi\ufb01cation problem de\ufb01ned over H is\ntakes D = {xi, c1\ni}n\ni=1 as input and outputs the solution to the optimization problem. We use\nCSC(H; D) to denote the classi\ufb01er returned by CSC(H) on data set D. We say that an algorithm\nis oracle ef\ufb01cient if it runs in polynomial time given the ability to make unit-time calls to CSC(H).\n\nthe optimization problem: arg minh\u2208H(cid:80)n\n\ni h(xi) + c0\n\ni , c0\n\ni (1 \u2212 h(xi))(cid:9). An oracle CSC(H)\n\ni , c0\n\ni and c0\n\n(cid:8)c1\n\ni=1\n\n3 Learning subject to AIF\n\nIn this section we \ufb01rst cast the learning problem subject to the AIF fairness constraints as the\nconstrained optimization problem (1) and then develop an oracle ef\ufb01cient algorithm for solving its\ncorresponding empirical risk minimization (ERM) problem (in the spirit of Agarwal et al. (2018)).\nIn the coming sections we give a full analysis of the developed algorithm including its in-sample\naccuracy/fairness guarantees and de\ufb01ne the mapping it induces from new problems to hypotheses,\nand \ufb01nally establish out-of-sample bounds for this trained mapping.\n\nFair Learning Problem subject to (\u03b1, 0)-AIF\n\nmin\n\n\u03c8 \u2208 \u2206(H)F , \u03b3 \u2208 [0,1]\ns.t. \u2200x \u2208 X :\n\nerr (\u03c8;P,Q)\n|E (x, \u03c8;Q) \u2212 \u03b3| \u2264 \u03b1\n\n(1)\n\nDe\ufb01nition 3.1 (OPT). Consider the optimization problem (1). Given distributions P and Q, and\nfairness approximation parameter \u03b1, we denote the optimal solutions of (1) by \u03c8(cid:63) (\u03b1;P,Q) and\n\u03b3(cid:63) (\u03b1;P,Q), and the value of the objective function at these optimal points by OPT (\u03b1;P,Q).\nWe will use OPT as the benchmark with respect to which we evaluate the accuracy of our trained\nmapping. It is worth noticing that the optimization problem (1) has a nonempty set of feasible\nsolutions for every \u03b1 and all distributions P and Q because the following point is always feasible:\n\u03b3 = 0.5 and \u03c8f = 0.5h0 + 0.5h1 (i.e. random classi\ufb01cation) for all f \u2208 F where h0 and h1 are\nall-zero and all-one constant classi\ufb01ers.\n\n3.1 The empirical fair learning problem\n\nWe start to develop our algorithm by de\ufb01ning the empirical version of (1) for a given training data\nset of n individuals X = {xi}n\nj=1. We will formulate the\n\ni=1 and m learning problems F = {fj}m\n\n4\n\n\fempirical problem as \ufb01nding a restricted mapping \u03c8|F by which we mean the domain of the mapping\nis restricted to the training set F \u2286 F. We will later see how the dynamics of our proposed algorithm\nallows us to extend the restricted mapping to a mapping from the entire space F. We slightly change\nnotation and represent a restricted mapping \u03c8|F explicitly by a vector p = (p1, . . . , pm) \u2208 \u2206(H)m\nof randomized classi\ufb01ers where pj \u2208 \u2206(H) corresponds to fj \u2208 F . Using the empirical versions of\nthe individual and the overall error rates incurred by the mapping p (see De\ufb01nition 2.1), we cast the\nempirical fair learning problem as the constrained optimization problem (2).\n\nEmpirical Fair Learning Problem\n\nmin\n\np \u2208 \u2206(H)m, \u03b3 \u2208 [0,1]\ns.t. \u2200i \u2208 {1, . . . , n}:\n\np;(cid:98)P, (cid:98)Q(cid:17)\n(cid:16)\n(cid:12)(cid:12)(cid:12)E(cid:16)\nxi, p; (cid:98)Q(cid:17) \u2212 \u03b3\n\nerr\n\n(cid:12)(cid:12)(cid:12) \u2264 2\u03b1\n\n(2)\n\nr\n\n(cid:16)\n\np, \u03b3; (cid:98)Q(cid:17)\n\nWe use the dual perspective of constrained optimization to reduce the fair learning task (2) to a\ntwo-player game between a \u201cLearner\u201d (primal player) and an \u201cAuditor\u201d (dual player). Towards\n\nderiving the Lagrangian of (2), we \ufb01rst rewrite its constraints in r (p, \u03b3; (cid:98)Q) \u2264 0 form where\n\n\uf8ee\uf8f0E(cid:16)\nxi, p; (cid:98)Q(cid:17) \u2212 \u03b3 \u2212 2\u03b1\n\u03b3 \u2212 E(cid:16)\nxi, p; (cid:98)Q(cid:17) \u2212 2\u03b1\n(cid:3)\nvariables for r be represented by \u03bb =(cid:2)\u03bb+\ni \u2208 \u039b, where \u039b = {\u03bb \u2208 R2n\n(2) is L (p, \u03b3, \u03bb) = err (p;(cid:98)P, (cid:98)Q) + \u03bbT r (p, \u03b3; (cid:98)Q). We therefore consider solving:\n\nrepresents the \u201cfairness violations\u201d of the pair (p, \u03b3) in one single vector. Let the corresponding dual\n+ |||\u03bb||1 \u2264 B}. Note\nwe place an upper bound B on the (cid:96)1-norm of \u03bb in order to reason about the convergence of our\nproposed algorithm. B will eventually factor into both the run-time and the approximation guarantees\nof our solution. Using Equation (3) and the introduced dual variables, we have that the Lagrangian of\n\n\uf8f9\uf8fbn\n\n\u2208 R2n\n\ni , \u03bb\n\n(3)\n\n\u2212\ni\n\ni=1\n\n=\n\nL (p, \u03b3, \u03bb)\n\nL (p, \u03b3, \u03bb) = max\n\u03bb\u2208\u039b\n\nmin\n\nmin\n\nmax\n\u03bb\u2208\u039b\n\np \u2208 \u2206(H)m, \u03b3 \u2208 [0,1]\n\np \u2208 \u2206(H)m, \u03b3 \u2208 [0,1]\n\n(4)\nwhere strong duality holds because L is linear in its arguments and the domains of (p, \u03b3) and \u03bb are\nconvex and compact (Sion (1958)). From a game theoretic perspective, the solution to this minmax\nproblem can be seen as an equilibrium of a zero-sum game between two players. The primal player\n(Learner) has strategy space \u2206(H)m \u00d7 [0, 1] while the dual player (Auditor) has strategy space \u039b,\nand given a pair of chosen strategies (p, \u03b3, \u03bb), the Lagrangian L (p, \u03b3, \u03bb) represents how much\nthe Learner has to pay to the Auditor \u2014 i.e. it de\ufb01nes the payoff function of a zero sum game in\nwhich the Learner is the minimization player, and the Auditor is the maximization player. Using\nno regret dynamics, an approximate equilibrium of this zero-sum game can be found in an iterative\nframework. In each iteration, we let the dual player run the exponentiated gradient descent algorithm\nand the primal player best respond. The best response problem of the Learner can be decoupled\ninto (m + 1) separate minimization problems and that in particular, the optimal classi\ufb01ers p can\nbe viewed as the solutions to m weighted classi\ufb01cation problems in H where all m problems share\ni ]i \u2208 Rn over the training individuals. We write the best response of the\nthe weights w = [\u03bb+\nLearner in Subroutine 1 where we use the oracle CSC(H) (see De\ufb01nition 2.3) to solve the weighted\nclassi\ufb01cation problems. See the supplementary \ufb01le for the detailed derivation.\n\ni \u2212 \u03bb\u2212\n\nSubroutine 1: BEST\u2013 best response of the Learner in the AIF setting\nInput: dual weights w = [\u03bb+\nfor j = 1, . . . , m do\n\ni=1 \u2208 Rn, training examples S =(cid:8)xi, (fj(xi))m\n\ni \u2212 \u03bb\u2212\ni ]n\n\nj=1\n\n(cid:9)n\n\ni=1\n\ni \u2190 (wi + 1/n)fj(xi) for i \u2208 [n].\ni , c0\n\ni}n\ni=1.\n\ni \u2190 (wi + 1/n)(1 \u2212 fj(xi)) and c0\nc1\nhj \u2190 CSC (H; D) where D = {xi, c1\n(cid:16)\n\nh = (h1, h2, . . . , hm) , \u03b3 = 1 [(cid:80)n\n\nend\nOutput:\n\n(cid:17)\n\ni=1 wi > 0]\n\n5\n\n\f3.2 Algorithm implementation and in-sample guarantees\n\nan average over T classi\ufb01ers where classi\ufb01er t is the solution to a CSC problem on X weighted\n\nIn Algorithm 2 (AIF-Learn), with a slight deviation from what we described in the previous sub-\nsection, we implement the proposed algorithm. The deviation arises when the Auditor updates the\ndual variables \u03bb in each round, and is introduced in the service of arguing for generalization. To\ncounteract the inherent adaptivity of the algorithm (which makes the quantities estimated at each\nround data dependent), at each round t of the algorithm, we draw a fresh batch of m0 problems. From\nanother viewpoint \u2013 which is the way the algorithm is actually implemented \u2013 similar to usual batch\nlearning models we assume we have a training set F of m learning problems upfront. However, in\nour proposed algorithm that runs for T iterations, we partition F into T equally-sized (m0) subsets\n{Ft}T\nt=1 uniformly at random and use only the batch Ft at round t to update \u03bb. Without loss of\nThis is represented in Algorithm 2 by writing (cid:98)Qt = U (Ft) for the uniform distribution over the batch\ngenerality and to avoid technical complications, we assume |Ft| = m0 = m/T is a natural number.\nof problems Ft , and ht|Ft for the associated classi\ufb01ers for Ft.\nNotice AIF-Learn takes as input an approximation parameter \u03bd \u2208 [0, 1] which will quantify how\nclose the output of the algorithm is to an equilibrium of the introduced game, and it will accordingly\nvector wt \u2208 Rn over the training individuals X and that each(cid:98)pj learned by our algorithm is in fact\npropagate to the accuracy bounds. One important aspect of AIF-Learn is that it maintains a weight\nby wt. As a consequence, we propose to extend the learned restricted mapping(cid:98)p to a mapping\n(cid:98)\u03c8 = (cid:98)\u03c8 (X,(cid:99)W ) that takes any problem f \u2208 F as input (represented to (cid:98)\u03c8 by the labels it induces on\nthe training data), uses the individuals X along with the set of weights(cid:99)W to solve T CSC problems\nin a similar fashion, and outputs the average of the learned classi\ufb01ers denoted by (cid:98)\u03c8f \u2208 \u2206(H). This\nextension is consistent with(cid:98)p in the sense that (cid:98)\u03c8 restricted to F will be exactly the(cid:98)p output by our\nalgorithm. The pseudocode for (cid:98)\u03c8 (output by AIF-Learn) is written in detail in Mapping 3.\nT , S \u2190(cid:8)xi, (fj(xi))j\n\n\u03b1 , T \u2190 16B2(1+2\u03b1)2 log(2n+1)\nt=1 where |Ft| = m0.\n\nAlgorithm 2: AIF-Learn \u2013 learning subject to AIF\nInput: fairness parameter \u03b1, approximation parameter \u03bd, data X = {xi}n\nB \u2190 1+2\u03bd\nPartition F : {Ft}T\n\u03b81 \u2190 0\nfor t = 1, . . . , T do\n1+(cid:80)\ni,t \u2190 B\n\u03bb\u2022\nwt \u2190 [\u03bb+\ni,t \u2212 \u03bb\u2212\n(ht, \u03b3t) \u2190 BEST(wt; S)\n\u03b8t+1 \u2190 \u03b8t + \u03b7 \u00b7 r\n\nfor i \u2208 [n] and \u2022 \u2208 {+,\u2212}\n\ni=1 and F = {fj}m\n\n, m0 \u2190 m\n\n, \u03b7 \u2190\n\n4(1+2\u03b1)2B\n\n(cid:17)\n\n(cid:9)\n\n\u03bd2\n\nj=1\n\ni\n\n\u03bd\n\n(cid:80)T\n\nend(cid:98)\u03b3 \u2190 1\n\nT\n\nt=1 \u03b3t ,\n\nOutput: average plays\n\n)\n\ni,t)\n\nexp(\u03b8\u2022\ni(cid:48) ,\u2022(cid:48) exp(\u03b8\u2022(cid:48)\ni(cid:48) ,t\ni=1 \u2208 Rn\n(cid:16)\ni,t]n\nht|Ft, \u03b3t; (cid:98)Qt\n(cid:80)T\n(cid:98)p \u2190 1\n(cid:17)\n(cid:16)(cid:98)p,(cid:98)\u03b3, (cid:98)\u03bb\n\nT\n\n(cid:80)T\n(cid:98)\u03bb \u2190 1\nt=1 \u03bbt , (cid:99)W \u2190 {wt}T\n(cid:16)\n, mapping (cid:98)\u03c8 = (cid:98)\u03c8\nX,(cid:99)W\n\n(see Mapping 3)\n\n(cid:17)\n\nT\n\nt=1\n\nt=1 ht ,\n\nWe defer a complete in-sample analysis of Algorithm 2 to the supplementary \ufb01le. At a high level, we\nstart by establishing the regret bound of the Auditor and choosing T and \u03b7 such that her regret \u2264 \u03bd.\n\nregret of the Auditor because she is using a batch of only m0 randomly selected problems to update\nthe fairness violation vector r. We therefore have to assume m0 is suf\ufb01ciently large to control the\n\nThere will be an extra (cid:101)O((cid:112)1/m0) term originating from a high probability (Chernoff) bound in the\nregret. Once the regret bound is established, we can show that the average played strategies ((cid:98)p,(cid:98)\u03b3, (cid:98)\u03bb)\nthis guarantee and turn it into accuracy and fairness guarantees of the pair ((cid:98)p,(cid:98)\u03b3) with respect to the\nempirical distributions ((cid:98)P, (cid:98)Q), which results in Theorem 3.1.\n\noutput by Algorithm 2 forms a \u03bd-approximate equilibrium of the game by which we mean: neither\nplayer would gain more than \u03bd if they deviated from these proposed strategies. Finally we can take\n\n6\n\n\fMapping 3: (cid:98)\u03c8 (X,(cid:99)W ) \u2013 pseudocode\n\nInput: f \u2208 F (represented as {f (xi)}n\nfor t = 1, . . . , T do\n\ni=1)\n\nT\n\nend\n\ni , c0\n\ni \u2190 (wi,t + 1/n)(1 \u2212 f (xi)) for i \u2208 [n].\nc1\ni \u2190 (wi,t + 1/n)f (xi) for i \u2208 [n].\nc0\ni}n\nD \u2190 {xi, c1\ni=1.\nhf,wt \u2190 CSC (H; D)\n(cid:80)T\nOutput: (cid:98)\u03c8f = 1\nt=1 hf,wt \u2208 \u2206(H)\nTheorem 3.1 (In-sample Accuracy and Fairness). Suppose m0 \u2265 O (log (nT /\u03b4)/\u03b12\u03bd2). Let ((cid:98)p,(cid:98)\u03b3)\nprobability 1 \u2212 \u03b4, err ((cid:98)p;(cid:98)P, (cid:98)Q) \u2264 err (p;(cid:98)P, (cid:98)Q) + 2\u03bd, and that(cid:98)p satis\ufb01es (3\u03b1, 0)-AIF with respect\nto the empirical distributions ((cid:98)P, (cid:98)Q). In other words, for all i \u2208 [n], |E(xi,(cid:98)p; (cid:98)Q) \u2212(cid:98)\u03b3| \u2264 3\u03b1.\n\nbe the output of Algorithm 2 and let (p, \u03b3) be any feasible pair of variables for (2). We have that with\n\nFigure 1: Illustration of generalization directions.\n\n3.3 Generalization theorems\n\nsimultaniously, given that both n and m are large enough. We will use OPT (see De\ufb01nition 3.1) as a\n\nwill remain accurate and fair with respect to Q. We will eventually put these pieces together in\n\nWhen it comes to out-of-sample performance in our framework, unlike in usual learning settings,\nthere are two distributions we need to reason about: the individual distribution P and the problem\ndistribution Q (see Figure 1 for a visual illustration of generalization directions in our framework).\nalmost every individual x \u223c P, where fairness is de\ufb01ned with respect to the true problem distribution\nQ. Given these two directions for generalization, we state our generalization guarantees in three\nsteps visualized by arrows in Figure 1. First, in Theorem 3.2, we \ufb01x the empirical distribution of\nunderlying individual distribution P as long as n is suf\ufb01ciently large. Second, in Theorem 3.3, we \ufb01x\n\nWe need to argue that (cid:98)\u03c8 induces a mapping that is accurate with respect to P and Q, and is fair for\nthe problems (cid:98)Q and show that the output (cid:98)\u03c8 of Algorithm 2 is accurate and fair with respect to the\nthe empirical distribution of individuals (cid:98)P and consider generalization along the underlying problem\ngenerating distribution Q. It will follow from the dynamics of the algorithm that the mapping (cid:98)\u03c8\nTheorem 3.4 and argue that (cid:98)\u03c8 is accurate and fair with respect to the underlying distributions (P,Q)\nbenchmark for the accuracy of the mapping (cid:98)\u03c8. See the supplementary \ufb01le for detailed proofs.\nTheorem 3.2 (Generalization over P). Let 0 < \u03b4 < 1. Let (cid:98)\u03c8 and(cid:98)\u03b3 be the outputs of Algorithm 2 and\nsuppose n \u2265 (cid:101)O(cid:0)(m dH + log (1/\u03bd2\u03b4))/\u03b12\u03b22(cid:1). We have that with probability 1\u2212 5\u03b4, the mapping (cid:98)\u03c8\nsatis\ufb01es (5\u03b1, \u03b2)-AIF with respect to the distributions (P, (cid:98)Q), i.e., Px \u223cP (|E(x,(cid:98)\u03c8; (cid:98)Q)\u2212(cid:98)\u03b3| > 5\u03b1) \u2264 \u03b2\nand that err ((cid:98)\u03c8;P, (cid:98)Q) \u2264 OPT (\u03b1;P, (cid:98)Q) + O (\u03bd) + O (\u03b1\u03b2) .\nTheorem 3.3 (Generalization over Q). Let 0 < \u03b4 < 1. Let (cid:98)\u03c8 and(cid:98)\u03b3 be the outputs of Algorithm 2 and\nsuppose m \u2265 (cid:101)O(cid:0) log (n) log (n/\u03b4) /\u03bd4\u03b14(cid:1). We have that with probability 1 \u2212 6\u03b4, the mapping (cid:98)\u03c8\nx \u223c(cid:98)P (|E(x,(cid:98)\u03c8;Q)\u2212(cid:98)\u03b3| > 4\u03b1) = 0\nsatis\ufb01es (4\u03b1, 0)-AIF with respect to the distributions ((cid:98)P,Q), i.e., P\nand that err ((cid:98)\u03c8;(cid:98)P,Q) \u2264 OPT (\u03b1;(cid:98)P,Q) + O (\u03bd).\nTheorem 3.4 (Simultaneous Generalization over P and Q). Let 0 < \u03b4 < 1. Let (cid:98)\u03c8 and (cid:98)\u03b3\nbe the outputs of Algorithm 2 and suppose n \u2265 (cid:101)O(cid:0)(m dH + log (1/\u03bd2\u03b4))/\u03b12\u03b22(cid:1) and m \u2265\n(cid:101)O(cid:0) log (n) log (n/\u03b4) /\u03bd4\u03b14(cid:1). We have that with probability 1\u221212\u03b4, the mapping(cid:98)\u03c8 satis\ufb01es (6\u03b1, 2\u03b2)-\nAIF with respect to the distributions (P,Q), i.e., Px \u223cP (|E(x,(cid:98)\u03c8;Q) \u2212(cid:98)\u03b3| > 6\u03b1) \u2264 2\u03b2 and that\nerr ((cid:98)\u03c8;P,Q) \u2264 OPT (\u03b1;P,Q) + O (\u03bd) + O (\u03b1\u03b2).\n\nNote that the bounds on n and m in Theorem 3.4 are mutually dependent: n must be linear in m, but\nm need only be logarithmic in n, and so both bounds can be simultaneously satis\ufb01ed with sample\ncomplexity that is only polynomial in the parameters of the problem.\n\n7\n\n\f4 Experimental evaluation\n\nFigure 2: (a) Error-unfairness trajectory plots illustrating the convergence of algorithm AIF-Learn.\n(b) In-sample error-unfairness tradeoffs and individual errors for AIF-Learn vs. the baseline model:\nsimple mixtures of the error-optimal model and random classi\ufb01cation. Gray dots are shifted upwards\nslightly to avoid occlusions.\n\nWe have implemented the AIF-Learn algorithm and conclude with a brief experimental demonstra-\ntion of its practical ef\ufb01cacy using the Communities and Crime dataset3, which contains U.S. census\nrecords with demographic information at the neighborhood level. To obtain a challenging instance of\nour multi-problem framework, we treated each of the \ufb01rst n = 200 neighborhoods as the \u201cindividuals\u201d\nin our sample, and binarized versions of the \ufb01rst m = 50 variables as distinct prediction problems.\nAnother d = 20 of the variables were used as features for learning. For the base learning oracle\nassumed by AIF-Learn, we used a linear threshold learning heuristic that has worked well in other\noracle-ef\ufb01cient reductions (Kearns et al. (2018)).\nDespite the absence of worst-case guarantees for the linear threshold heuristic, AIF-Learn seems\nto empirically enjoy the strong convergence properties suggested by the theory. In Figure 2(a) we\nshow trajectory plots of the learned model\u2019s error (x axis) versus its fairness violation (variation in\ncross-problem individual error rates, y axis) over 1000 iterations of the algorithm for varying values\nof the allowed fairness violation 2\u03b1 (dashed horizontal lines). In each case we see the trajectory\neventually converge to a point which saturates the fairness constraint with the optimal error.\nIn Figure 2(b) we provide a more detailed view of the behavior and performance of AIF-Learn.\nThe x axis measures error rates, while the y axis measures the allowed fairness violation. For each\nvalue of the allowed fairness violation 2\u03b1 (which is the allowed gap between the smallest and largest\nindividual errors on input \u03b1), there is a horizontal row of 200 blue dots showing the error rates for\neach individual, and a single red dot representing the overall average of those individual error rates.\nAs expected, for large \u03b1 (weak or no fairness constraint), the overall error rate is lowest, but the\nspread of individual error rates (unfairness) is greatest. As \u03b1 is decreased, the spread of individual\nerror rates is greatly narrowed, at a cost of greater overall error.\n\nA trivial way of achieving zero variability in individual error rates is to make all predictions randomly.\nSo as a baseline comparison for AIF-Learn, the gray dots in Figure 2(b) show the individual\nerror rates achieved by different mixtures of the unconstrained error-optimal model with random\nclassi\ufb01cations, with a black dot representing the overall average of these rates. When the weight on\nrandom classi\ufb01cation is low (weak or no fairness, top row of gray dots), the overall error is lowest and\nthe individual variation (unfairness) is highest. As we increase the weight on random classi\ufb01cation,\nvariation or unfairness decreases and the overall error gets worse. It is clear from the \ufb01gure that\nAIF-Learn is considerably outperforming this baseline, both in terms of the average errors (red vs.\nblack lines) and the individual errors (blue vs. gray dots).\n\n3Described in detail and available for download at http://archive.ics.uci.edu/ml/datasets/\n\ncommunities+and+crime\n\n8\n\n\fFigure 3: Pareto frontier of error and fairness violation rates on training and test data sets.\n\nFinally we present out-of-sample performance of AIF-Learn in Figure 3. To be consistent with\nin-sample results reported in Figure 2(b), for each value of \u03b1, we trained a mapping on exactly the\nsame subset of the Communities and Crime data set (n = 200 individuals, m = 50 problems) that\nwe used before. Thus the red curve labelled \u201ctraining\u201d in Figure 3 is the same as the red curve\nappearing in Figure 2(b). We used a completely fresh holdout consisting of n = 200 individuals and\nm = 25 problems (binarized features from the dataset that weren\u2019t previously used) to evaluate our\ngeneralization performance over both individuals and problems, in terms of both accuracy and fairness\nviolation. Similar to the presentation of generalization theorems in Section 3.3, we demonstrate\nexperimental evaluation of generalization in three steps. The blue and green curves in Figure 3\nrepresent generalization results over individuals (test data: test individuals and training problems) and\nproblems (test data: training individuals and test problems) respectively. The black curve represent\ngeneralization across both individuals and problems where test individuals and test problems were\nused to evaluate the performance of the trained models.\nTwo things stand out from Figure 3:\n\n1. As predicted by the theory, our test curves track our training curves, but with higher error\nand unfairness. In particular, the ordering of the models (each corresponds to one \u03b1) on\nthe Pareto frontier is the same in testing as in training, meaning that the training curve can\nindeed be used to manage the trade-off out-of-sample as well.\n\n2. The gap in error is substantially smaller than would be predicted by our theory: since our\ntraining data set is so small, our theoretical guarantees are vacuous, but all points plotted in\nour test Pareto curves are non-trivial in terms of both accuracy and fairness. Presumably the\ngap in error would narrow on larger training data sets.\n\nWe have additional experimental results on a synthetic data set in the supplement.\n\nReferences\nAbdulkadiro\u02d8glu, A., Pathak, P. A., and Roth, A. E. (2005). The new york city high school match.\n\nAmerican Economic Review, 95(2):364\u2013367.\n\nAgarwal, A., Beygelzimer, A., Dudik, M., Langford, J., and Wallach, H. (2018). A reductions ap-\nproach to fair classi\ufb01cation. In Dy, J. and Krause, A., editors, Proceedings of the 35th International\nConference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages\n60\u201369, Stockholmsm\u00e4ssan, Stockholm Sweden. PMLR.\n\n9\n\n\fChouldechova, A. (2017). Fair prediction with disparate impact: A study of bias in recidivism\n\nprediction instruments. Big data, 5(2):153\u2013163.\n\nDwork, C., Hardt, M., Pitassi, T., Reingold, O., and Zemel, R. (2012). Fairness through awareness.\nIn Proceedings of the 3rd innovations in theoretical computer science conference, pages 214\u2013226.\nACM.\n\nGillen, S., Jung, C., Kearns, M., and Roth, A. (2018). Online learning with an unknown fairness\n\nmetric. In Advances in Neural Information Processing Systems, pages 2600\u20132609.\n\nHardt, M., Price, E., Srebro, N., et al. (2016). Equality of opportunity in supervised learning. In\n\nAdvances in neural information processing systems, pages 3315\u20133323.\n\nH\u00e9bert-Johnson, \u00da., Kim, M. P., Reingold, O., and Rothblum, G. N. (2018). Multicalibration: Calibra-\ntion for the (computationally-identi\ufb01able) masses. In Dy, J. G. and Krause, A., editors, Proceedings\nof the 35th International Conference on Machine Learning, ICML 2018, Stockholmsm\u00e4ssan, Stock-\nholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pages\n1944\u20131953. PMLR.\n\nJoseph, M., Kearns, M., Morgenstern, J., Neel, S., and Roth, A. (2018). Meritocratic fairness for\nin\ufb01nite and contextual bandits. In Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics,\nand Society, pages 158\u2013163. ACM.\n\nJoseph, M., Kearns, M., Morgenstern, J. H., and Roth, A. (2016). Fairness in learning: Classic and\n\ncontextual bandits. In Advances in Neural Information Processing Systems, pages 325\u2013333.\n\nKearns, M., Neel, S., Roth, A., and Wu, Z. S. (2018). Preventing fairness gerrymandering: Auditing\nand learning for subgroup fairness. In International Conference on Machine Learning, pages\n2569\u20132577.\n\nKearns, M., Neel, S., Roth, A., and Wu, Z. S. (2019). An empirical study of rich subgroup fairness for\nmachine learning. In Proceedings of the Conference on Fairness, Accountability, and Transparency,\npages 100\u2013109. ACM.\n\nKim, M., Reingold, O., and Rothblum, G. (2018a). Fairness through computationally-bounded\n\nawareness. In Advances in Neural Information Processing Systems, pages 4842\u20134852.\n\nKim, M. P., Ghorbani, A., and Zou, J. Y. (2018b). Multiaccuracy: Black-box post-processing for\n\nfairness in classi\ufb01cation. CoRR, abs/1805.12317.\n\nKleinberg, J., Mullainathan, S., and Raghavan, M. (2017). Inherent trade-offs in the fair determination\nof risk scores. In 8th Innovations in Theoretical Computer Science Conference (ITCS 2017),\nvolume 67, page 43. Schloss Dagstuhl\u2013Leibniz-Zentrum fuer Informatik.\n\nMitchell, S., Potash, E., and Barocas, S. (2018). Prediction-based decisions and fairness: A catalogue\n\nof choices, assumptions, and de\ufb01nitions. arXiv preprint arXiv:1811.07867.\n\nSion, M. (1958). On general minimax theorems. Paci\ufb01c J. Math., 8(1):171 \u2013 176.\n\nYona, G. and Rothblum, G. N. (2018). Probably approximately metric-fair learning. In Dy, J. G.\nand Krause, A., editors, Proceedings of the 35th International Conference on Machine Learning,\nICML 2018, Stockholmsm\u00e4ssan, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings\nof Machine Learning Research, pages 5666\u20135674. PMLR.\n\n10\n\n\f", "award": [], "sourceid": 4474, "authors": [{"given_name": "Saeed", "family_name": "Sharifi-Malvajerdi", "institution": "University of Pennsylvania"}, {"given_name": "Michael", "family_name": "Kearns", "institution": "University of Pennsylvania"}, {"given_name": "Aaron", "family_name": "Roth", "institution": "University of Pennsylvania"}]}