{"title": "Enhancing the Accuracy and Fairness of Human Decision Making", "book": "Advances in Neural Information Processing Systems", "page_first": 1769, "page_last": 1778, "abstract": "Societies often rely on human experts to take a wide variety of decisions affecting their members, from jail-or-release decisions taken by judges and stop-and-frisk decisions taken by police officers to accept-or-reject decisions taken  by academics. In this context, each decision is taken by an expert who is typically chosen uniformly at random from a pool of experts. However, these decisions may be imperfect due to limited experience, implicit biases, or faulty probabilistic reasoning. Can we improve the accuracy and fairness of the overall decision making process by optimizing the assignment between experts and decisions?\n\nIn this paper, we address the above problem from the perspective of sequential decision making and show that, for different fairness notions from the literature, it reduces to a sequence of (constrained) weighted bipartite matchings, which can be solved efficiently using algorithms with approximation guarantees. Moreover, these algorithms also benefit from posterior sampling to actively trade off exploitation---selecting expert assignments which lead to accurate and fair decisions---and exploration---selecting expert assignments to learn about the experts' preferences and biases. We demonstrate the effectiveness of our algorithms on both synthetic and real-world data and show that they can significantly improve both the accuracy and fairness of the decisions taken by pools of experts.", "full_text": "Enhancing the Accuracy and Fairness\n\nof Human Decision Making\n\nIsabel Valera\u2217\n\nMPI for Intelligent Systems\n\nivalera@tue.mpg.de\n\nAdish Singla\u2020\nMPI-SWS\n\nadishs@mpi-sws.org\n\nAbstract\n\nManuel Gomez-Rodriguez\u2021\n\nMPI-SWS\n\nmanuelgr@mpi-sws.org\n\nSocieties often rely on human experts to take a wide variety of decisions affecting\ntheir members, from jail-or-release decisions taken by judges and stop-and-frisk\ndecisions taken by police of\ufb01cers to accept-or-reject decisions taken by academics.\nIn this context, each decision is taken by an expert who is typically chosen uniformly\nat random from a pool of experts. However, these decisions may be imperfect\ndue to limited experience, implicit biases, or faulty probabilistic reasoning. Can\nwe improve the accuracy and fairness of the overall decision making process by\noptimizing the assignment between experts and decisions?\nIn this paper, we address the above problem from the perspective of sequential\ndecision making and show that, for different fairness notions in the literature, it\nreduces to a sequence of (constrained) weighted bipartite matchings, which can\nbe solved ef\ufb01ciently using algorithms with approximation guarantees. More-\nover, these algorithms also bene\ufb01t from posterior sampling to actively trade\noff exploitation\u2014selecting expert assignments which lead to accurate and fair\ndecisions\u2014and exploration\u2014selecting expert assignments to learn about the ex-\nperts\u2019 preferences. We demonstrate the effectiveness of our algorithms on both\nsynthetic and real-world data and show that they can signi\ufb01cantly improve both the\naccuracy and fairness of the decisions taken by pools of experts.\n\nIntroduction\n\n1\nIn recent years, there have been increasing concerns about the potential for unfairness of algorithmic\ndecision making. Moreover, these concerns have been often supported by empirical studies, which\nhave provided, e.g., evidence of racial discrimination [8, 10]. As a consequence, there have been\na \ufb02urry of work on developing computational mechanisms to make sure that the machine learning\nmethods that fuel algorithmic decision making are fair [3, 4, 5, 6, 13, 14, 15]. In contrast, to the\nbest of our knowledge, there is a lack of machine learning methods to ensure accuracy and fairness\nin human decision making, which is still prevalent in a wide range of critical applications such as,\ne.g., jail-or-release decisions by judges, stop-and-frisk decisions by police of\ufb01cers or accept-or-reject\ndecisions by academics. In this work, we take a \ufb01rst step towards \ufb01lling this gap.\nMore speci\ufb01cally, we focus on a problem setting that \ufb01ts a variety of real-world applications, including\nthe ones mentioned above: binary decisions come sequentially over time and each decision need to\nbe taken by a human decision maker, typically an expert, who is chosen from a pool of experts. For\nexample, in jail-or-release decisions, the expert is a judge who needs to decide whether she grants\nbail to a defendant; in stop-and-frisk decisions, the expert is a police of\ufb01cer who needs to decide\nwhether she stop (and potentially frisk) a pedestrian; or, in accept-or-reject decisions, the expert is\nan academic who needs to decide whether a paper is accepted in a conference (or a journal). In\nthis context, our goal is then to \ufb01nd the optimal assignments between human decision makers and\n\n\u2217Max Planck Institute for Intelligent Systems. Max Planck Ring 4, 472076 Tuebingen (Germany).\n\u2020Max Planck Institute for Software Systems (MPI-SWS). Campus E1 5, 66123 Saarbruecken (Germany).\n\u2021Max Planck Institute for Software Systems. Paul-Ehrlich-Strasse, G26, 67663 Kaiserslautern (Germany).\n\n32nd Conference on Neural Information Processing Systems (NeurIPS 2018), Montr\u00e9al, Canada.\n\n\fdecisions which maximizes the accuracy of the overall decision making process while satisfying\nseveral popular notions of fairness studied in the literature.\nIn this paper, we represent human decision making using threshold decisions rules [3] and then\nshow that, if the thresholds used by each expert are known, the above problem can be reduced to\na sequence of matching problems, which can be solved ef\ufb01ciently with approximation guarantees.\nMore speci\ufb01cally:\n\nI. Under no fairness constraints, the problem can be cast as a sequence of maximum weighted\nbipartite matching problems, which can be solved exactly in polynomial (quadratic)\ntime [12].\n\nII. Under (some of the most popular) fairness constraints, the problem can be cast as a sequence\nof bounded color matching problems, which can be solved using a bi-criteria algorithm\nbased on linear programming techniques with a 1/2 approximation guarantee [9].\n\nMoreover, if the thresholds used by each expert are unknown, we also show that, if we estimate the\nvalue of each threshold using posterior sampling, we can effectively trade off exploitation\u2014taking\naccurate and fair decisions\u2014and exploration\u2014learning about the experts\u2019 preferences and biases.\nMore formally, we can show that posterior samples achieve a sublinear regret in contrast to point\nestimates, which suffer from linear regret.\nFinally, we experiment on synthetic data and real jail-or-release decisions by judges [8]. The results\nshow that: (i) our algorithms improve the accuracy and fairness of the overall human decision making\nprocess with respect to random assignment; (ii) our algorithms are able to ensure fairness more\neffectively if the pool of experts is diverse, e.g., there exist harsh judges, lenient judges, and judges in\nbetween; and, (iii) our algorithms are able to ensure fairness even if a signi\ufb01cant percentage of judges\n(e.g., 50%) has preferences (biases) against a group of individuals sharing a certain sensitive attribute\nvalue (e.g., race). The implementations of our algorithms and the data used in our experiments are\navailable at https://github.com/Networks-Learning/FairHumanDecisions.\n2 Preliminaries\nIn this section, we \ufb01rst de\ufb01ne decision rules and formally de\ufb01ne their utility and group bene\ufb01t. Then,\nwe revisit threshold decision rules, a type of decision rules which are optimal in terms of accuracy\nunder several notions of fairness from the literature.\nDecision rules, their utilities, and their group bene\ufb01ts. Given an individual with a feature vector\nx \u2208 Rd, a (ground-truth) label y \u2208 {0, 1}, and a sensitive attribute z \u2208 {0, 1}, a decision rule\nd(x, z) \u2208 {0, 1} controls whether the ground-truth label y is realized by means of a binary decision\nabout the individual. As an example, in a pretrial release scenario, the decision rule speci\ufb01es whether\nthe individual remains in jail, i.e., d(x, z) = 1 if she remains in jail and d(x, z) = 0 otherwise; the\nlabel indicates whether a released individual would reoffend, i.e., y = 1 if she would reoffend and\ny = 0 otherwise; the feature vector x may include the current offense, previous offenses, or times\nshe failed to appear in court; and the sensitive attribute z may be race, i.e., black vs white.\nFurther, we de\ufb01ne random variables X, Y , and Z that take on values X = x, Y = y, and Z = z for\nan individual drawn randomly from the population of interest. Then, we measure the (immediate)\nutility as the overall pro\ufb01t obtained by the decision maker using the decision rule [3], i.e.,\n\nu(d, c) = E [Y d(X, Z) \u2212 c d(X, Z)] = E(cid:2)d(X, Z)(cid:0)PY |X,Z \u2212 c(cid:1)(cid:3)\n\n(1)\nwhere c \u2208 (0, 1) is a given constant. For example, in a pretrial release scenario, the \ufb01rst term\nis proportional to the expected number of violent crimes prevented under d, the second term is\nproportional to the expected number of people detained, and c measures the cost of detention in units\nof crime prevented. Here, note that the above utility re\ufb02ects only the proximate costs and bene\ufb01ts of\ndecisions rather than long-term, systematic effects. Finally, we de\ufb01ne the (immediate) group bene\ufb01t\nas the fraction of bene\ufb01cial decisions received by a group of individuals sharing the sensitive attribute\nvalue z [15], i.e.,\n(2)\nFor example, in a pretrial release scenario, one may de\ufb01ne f (x) = 1 \u2212 x and thus the bene\ufb01t to the\ngroup of white individuals be proportional to the expected number of them who are released under d.\nRemarkably, most of the notions of (un)fairness used in the literature, such as disparate impact [1],\nequality of opportunity [6] or disparate mistreatment [13] can be expressed in terms of group bene\ufb01ts.\nFinally, note that, in some applications, the bene\ufb01cial outcome may correspond to d(X, Z) = 1.\n\nbz(d, c) = E [f (d(X, Z = z))] .\n\n2\n\n\f(cid:26)1\n\n0\n\n(cid:26)1\n\n0\n\n(cid:26)1\n\n0\n\nOptimal threshold decision rules. Assume the conditional distribution P (Y |X, Z) is given4. Then,\nthe optimal decision rules d\u2217 that maximize u(d, c) under the most popular fairness constraints from\nthe literature are threshold decision rules [3, 6]:\n\n\u2014 No fairness constraints: the optimal decision rule under no fairness constraints is given by\n\nthe following deterministic threshold rule:\n\nd\u2217(X, Z) =\n\nif pY =1|X,Z \u2265 c\notherwise.\n\n(3)\n\n\u2014 Disparate impact, equality of opportunity, and disparate mistreatment: the optimal decision\nrule which satis\ufb01es (avoids) the three most common notions of (un)fairness is given by the\nfollowing deterministic threshold decision rule:\n\nd\u2217(X, Z) =\n\nif pY =1|X,Z \u2265 \u03b8Z\notherwise,\n\n(4)\n\nwhere \u03b8Z \u2208 [0, 1] are constants that depend only on the sensitive attribute and the fairness\nnotion of interest. Note that the unconstraint optimum can be also expressed using the above\nform if we take \u03b8Z = c.\n3 Problem Formulation\nIn this section, we \ufb01rst use threshold decision rules to represent biased human decisions and then\nformally de\ufb01ne our sequential human decision making process.\nHumans as threshold decision rules. Inspired by recent work by Kleinberg et al. [7], we model a\nhuman decision maker v who has access to pY |X,Z using the following threshold decision rule:\n\ndv(X, Z) =\n\nif pY =1|X,Z \u2265 \u03b8V,Z\notherwise,\n\n(5)\n\nwhere \u03b8V,Z \u2208 [0, 1] are constants that depend on the decision maker and the sensitive attribute, and\nthey represent human decision makers\u2019 biases (or preferences) towards groups of people sharing\na certain value of the sensitive attribute z. For example, in a pretrial release scenario, if a judge v\nis generally more lenient towards white people (z = 0) than towards black people (z = 1), then\n\u03b8v,z=0 > \u03b8v,z=1.\nIn the above formulation, note that we assume all experts make predictions using the same (true)\nconditional distribution pY |X,Z, i.e., all experts have the same prediction ability.\nIt would be\nvery interesting to relax this assumption and account for experts with different prediction abilities.\nHowever, this entails a number of non trivial challenges and is left for future work.\nSequential human decision making problem. A set of human decision makers V = {vk}k\u2208[n] need\nto take decisions about individuals over time. More speci\ufb01cally, at each time t \u2208 {1, . . . , T}, there\nare m decisions to be taken and each decision i \u2208 [m] is taken by a human decision maker vi(t) \u2208 V,\nwho applies her threshold decision rule dvi(t)(X, Z), de\ufb01ned by Eq. 5, to the corresponding feature\nvector xi(t) and sensitive attribute zi(t). Note that we assume that yi(t)|xi(t), zi(t) \u223c pY |X,Z for\nall t \u2208 {1, . . . , T} and i \u2208 [m].\nAt each time t, our goal is then to \ufb01nd the assignment of human decision makers vi(t) to individuals\n(xi(t), zi(t)), with vi(t) (cid:54)= vj(t) for all i (cid:54)= j, that maximizes the expected utility of a sequence of\ndecisions, i.e.,\n\nu\u2264T ({dvi(t)}i,t, c) =\n\n1\nmT\n\ndvi(t)(xi(t), zi(t))(pY =1|xi(t),zi(t) \u2212 c),\n\n(6)\n\nwhere u\u2264T ({dvi(t)}i,t, c) is a empirical estimate of a straight forward generalization of the utility\nde\ufb01ned by Eq. 1 to multiple decision rules.\n\n4In practice, the conditional distribution may be approximated using a machine learning model trained on\n\nhistorical data.\n\n3\n\nT(cid:88)\n\nm(cid:88)\n\nt=1\n\ni=1\n\n\f4 Proposed Algorithms\nIn this section, we formally address the problem de\ufb01ned in the previous section without and with\nfairness constraints. Note that, without fairness constraints, we aim to approximate the solution\nprovided by Eq. 3, and with fairness constraints, Eq. 4. In both cases, we \ufb01rst consider the setting in\nwhich the human decision makers\u2019 thresholds \u03b8V,Z are known and then generalize our algorithms to\nthe setting in which they are unknown and need to be learned over time.\nDecisions under no fairness constraints. We can \ufb01nd the assignment of human decision makers\n{v(t)}T\n\nt=1 with the highest expected utility by solving the following optimization problem:\n\nT(cid:88)\n\nm(cid:88)\n\ndvi(t)(xi(t), zi(t))(pY =1|xi(t),zi(t) \u2212 c)\n\nmaximize\nsubject to vi(t) \u2208 V for all t \u2208 {1, . . . , T},\n\nt=1\n\ni=1\n\nvi(t) (cid:54)= vj(t) for all i (cid:54)= j.\n\n(7)\n\n\u2014 Known thresholds. If the thresholds \u03b8V,Z are known for all human decision makers, the above\nproblem decouples into T independent subproblems, one per time t \u2208 {1, . . . , T}, and each of these\nsubproblems can be cast as a maximum weighted bipartite matching, which can be solved exactly\nin polynomial (quadratic) time [12]. To do so, for each time t, we build a weighted bipartite graph\nwhere each human decision maker vj is connected to each individual (xi(t), zi(t)) with weigh:\n\n(cid:26)pY |xi(t),zi(t) \u2212 c\n\nwji =\n\n0\n\nif pY |xi(t),zi(t) \u2265 \u03b8vj ,zi(t)\notherwise,\n\nNote that the maximum weighted bipartite matching is the optimal assignment as de\ufb01ned by Eq. 7.\n\u2014 Unknown thresholds. If the thresholds are unknown, we need to trade off exploration, i.e., learning\nabout the thresholds \u03b8V,Z, and exploitation, i.e., maximizing the average utility. To this aim, for\nevery decision maker v, we assume a Beta prior over each threshold \u03b8v,z \u223c Beta(\u03b1, \u03b2). Under this\nassumption, after round t, we can update the (domain of the) distribution of \u03b8v,z(t) as:\n\nmax(0, \u03b8L\n\nv,z(t)) \u2264 \u03b8v,z(t) \u2264 min(1, \u03b8H\n\nv,z(t)),\n\nwhere\n\n\u03b8L\nv,z(t) =\n\n\u03b8H\nv,z(t) =\n\nt(cid:48)\u2264t | zi(t(cid:48))=z, vi(t(cid:48))=v, dv(xi(t(cid:48)),z)=0\n\nmax\n\npY =1|xi(t(cid:48)),z\n\nt(cid:48)\u2264t | zi(t(cid:48))=z, vi(t(cid:48))=v, dv(xi(t(cid:48)),z)=1\n\nmin\n\npY =1|xi(t(cid:48)),z,\n\nand write the posterior distribution of \u03b8v,z(t) as\n\np(\u03b8v,z(t)|D(t)) =\n\n\u0393(\u03b1 + \u03b2)(\u03b8H\n\nv,z(t) \u2212 \u03b8v,z(t))\u03b1\u22121(\u03b8v,z(t) \u2212 \u03b8L\nv,z(t))\u03b1+\u03b2\u22121\n\u0393(\u03b1)\u0393(\u03b2)(\u03b8H\n\nv,z(t) \u2212 \u03b8L\n\nv,z(t))\u03b2\u22121\n\n(8)\n\n,\n\n(9)\n\nThen, at the beginning of round t + 1, one can think of estimating the value of each threshold \u03b8v,z(t)\nusing point estimates, i.e., \u02c6\u03b8v,z = argmax p(\u03b8v,z(t)|D(t)), and use the same algorithm as for known\nthresholds. Unfortunately, if we de\ufb01ne regret as follows:\n\nR(T ) = u\u2264T ({dvi(t)}i,t, c) \u2212 u\u2264T ({dv\u2217\n\ni (t)}i,t, c),\n\nwhere vi(t) is the optimal assignment under the point estimates of the thresholds and v\u2217\noptimal assignment under the true thresholds, we can show that (proven in Appendix A):\n\n(10)\ni (t) is the\n\nProposition 1 The optimal assignments with deterministic point estimates for the thresholds suffers\nlinear regret \u0398(T ).\n\nThe above result is a consequence of insuf\ufb01cient exploration, which we can overcome if we esti-\nmate the value of each threshold \u03b8v,z(t) using posterior sampling, i.e., \u02c6\u03b8v,z \u223c p(\u03b8v,z(t)|D(t)), as\nformalized by the following theorem:\n\n\u221a\nTheorem 2 The expected regret of the optimal assignments with posterior samples for the thresholds\nis O(\n\nT ).\n\n4\n\n\fProof Sketch. The proof of this theorem follows via interpreting the problem setting as a reinforcement\nlearning problem. Then, we can apply the generic results for reinforcement learning via posterior\nsampling of [11]. In particular, we map our problem to an MDP with horizon 1 as follows. The\nactions in the MDP correspond to assigning m individuals to n experts (given by K) and the reward\nis given by the utility at time t.\nThen, it is easy to conclude that the expected regret of the optimal assignments with posterior samples\n\nfor the thresholds is O(S(cid:112)KT log(SKT )), where K = n.(n \u2212 1).(n \u2212 2) . . . (n \u2212 m + 1) denotes\n\nthe possible assignments of m individuals to n experts and S is a problem dependent parameter. S\nquanti\ufb01es the the total number of states/realizations of feature vectors xi and sensitive features zi to\nthe i \u2208 [m] individuals\u2014note that S is bounded only for the setting where feature vectors xi and\nsensitive features zi are discrete.\n\nGiven that the regret only grows as O(\nT ) (i.e., sublinear in T ), this theorem implies that the\nalgorithm based on optimal assignments with posterior samples converges to the optimal assignments\ngiven the true thresholds as T \u2192 \u221e.\nDecisions under fairness constraints. For ease of exposition, we focus on disparate impact, however,\na similar reasoning follows for equality of opportunity and disparate mistreatment [6, 13].\nTo avoid disparate impact, the optimal decision rule d\u2217(X, Z), given by Eq. 4, maximizes the utility,\nas de\ufb01ned by Eq. 1, under the fairness constraint [3, 13] DI(d\u2217, c) = |bz=1(d\u2217, c)\u2212bz=0(d\u2217, c)| \u2264 \u03b1,\nwhere \u03b1 \u2208 [0, 1] is a given parameter which controls the amount of disparate impact\u2014the smaller\nthe value of \u03b1, the lower the disparate impact of the corresponding decision rule. Similarly, we can\ncalculate a empirical estimate of the disparate impact of a decision rule d at each time t as:\n\n\u221a\n\n1\nm\n\n|bt,z=1(d, c) \u2212 bt,z=0(d, c)| ,\n\nwhere bt,z(d, c) =(cid:80)m\nconverges to DI(d\u2217, c) as m \u2192 \u221e, and 1/T(cid:80)T\n\n(11)\nDIt(d, c) =\nI(zi = z)f (d(xi(t), zi(t))), where f (\u00b7) de\ufb01nes what is a bene\ufb01cial out-\ncome. Here, it is easy to see that, for the optimal decision rule d\u2217 under impact parity, DIt(d\u2217, c)\nt=1 DIt(d\u2217, c) converges to DI(d\u2217, c) as T \u2192 \u221e.\nFor a \ufb01xed \u03b1, assume there are at least m(1 \u2212 \u03b1) experts with \u03b8v,z < c, at least m(1 \u2212 \u03b1) experts\nwith \u03b8v,z \u2265 c for each z = 0, 1, and n \u2265 2m. Then, we can \ufb01nd the assignment of human decision\nmakers {v(t)} with the highest expected utility and disparate impact less than \u03b1 as:\n\ni=1\n\nT(cid:88)\n\nm(cid:88)\n\ndvi(t)(xi(t), zi(t))(pY =1|xi(t),zi(t) \u2212 c),\n\nmaximize\nsubject to vi(t) \u2208 V for all t \u2208 {1, . . . , T},\n\nt=1\n\ni=1\n\nvi(t) (cid:54)= vj(t) for all i (cid:54)= j,\nbt,z(d\u2217, c) \u2212 \u03b1mz(t) \u2264 bt,z({dvi(t)}i)\u2200 t, z\nbt,z({dvi(t)}i) \u2264 bt,z(d\u2217, c) + \u03b1mz(t)\u2200 t, z.\n\ni=1\n\n(12)\n(cid:80)m\nwhere and mz(t) is the number of decisions with sensitive attribute z at round t and bt,z({dvi(t)}i) =\nI(zi = z)f (dvi(t)(xi(t), zi(t)))). Here, the assignment v\u2217(t) given by the solution to the\nabove optimization problem satis\ufb01es that DIt({dv\u2217\ni (t)}i, c) \u2208 [DIt(d\u2217, c) \u2212 \u03b1, DIt(d\u2217, c) + \u03b1] and\nthus limT\u2192\u221e DI\u2264T ({dv\u2217\n\u2014 Known thresholds. If the thresholds are known, the problem decouples into T independent\nsubproblems, one per time t \u2208 {1, . . . , T}, and each of these subproblems can be cast as a constrained\nmaximum weighted bipartite matching. To do so, for each time t, we build a weighted bipartite graph\nwhere each human decision maker vj is connected to each individual (xi(t), zi(t)) with weight wji,\nwhere\n\ni (t)}i,t, c) \u2264 \u03b1.\n\n(cid:26)pY |xi(t),zi(t) \u2212 c\n\nif pY |xi(t),zi(t) \u2265 \u03b8vj ,zi(t)\notherwise,\n\nand we additionally need to ensure that, for z \u2208 {0, 1}, the matching S satis\ufb01es that\n\nwji =\n\n0\n\nbt,z(d\u2217, c) \u2212 \u03b1mz(t) \u2264 (cid:88)\n\ng(wji)\n\n(j,i)\u2208S:zi=z\n\nand (cid:88)\n\n(j,i)\u2208S:zi=z\n\n5\n\ng(wji) \u2264 bt,z(d\u2217, c) + \u03b1mz(t),\n\n\fwhere mz(t) denotes the number of individuals with sensitive attribute z at round t and the function\ng(wji) depends on what is the bene\ufb01cial outcome, e.g., in a pretrial release scenario, g(wji) =\nI(wji (cid:54)= 0). Remarkably, we can reduce the above constrained maximum weighted bipartite matching\nproblem to an instance of the bounded color matching problem [9], which allows for a bi-criteria\nalgorithm based on linear programming techniques with a 1/2 approximation guarantee. To do so,\nwe just need to rewrite the above constraints as\n\ng(wji) \u2264 bt,z(d\u2217, c) + \u03b1mz(t),\n\nand\n\n(13)\n\n(cid:88)\n(cid:88)\n\n(j,i)\u2208S:zi=z\n\ngC(wji) \u2264 (1 + \u03b1)mz(t) \u2212 bt,z(d\u2217, c).\n\n(14)\n\n(j,i)\u2208S:zi=z\n\nwe are looking for a perfect matching and thus(cid:80)\n\nTo see the equivalence between the above constraints and the original ones, one needs to realize that\nexample, in a pretrial release scenario, g(wji) = I(wji (cid:54)= 0) and gC(wji) = I(wji = 0).\n\u2014 Unknown thresholds. If the threshold are unknown, we proceed similarly as in the case under\nno fairness constraints, i.e., we again assume Beta priors over each threshold, update their posterior\ndistributions after each time t, and use posterior sampling to set their values at each time.\nFinally, for the regret analysis, we focus on an alternative unconstrained problem, which is equivalent\nto the one de\ufb01ned by Eq. 12 by Lagrangian duality [2]:\n\n(cid:2)g(wji) + gC(wji)(cid:3) = mz(t). For\n\n(j,i)\u2208S:zi=z\n\nt=1\n\nT(cid:88)\nm(cid:88)\nm(cid:88)\n\ni=1\n\ni=1\n\ndvi(t)(xi(t), zi(t))(pY =1|xi(t),zi(t) \u2212 c)\n\nm(cid:88)\n(cid:0)bt,z(d\u2217, c) \u2212 bt,z({dvi(t)}i) \u2212 \u03b1mz(t)(cid:1)\n(cid:0)bt,z({dvi(t)}i) \u2212 bt,z(d\u2217, c) \u2212 \u03b1mz(t)(cid:1)\n\n\u03bbl,t,z\n\n\u03bbu,t,z\nvi(t) \u2208 V for all t \u2208 {1, . . . , T},\nvi(t) (cid:54)= vj(t) for all i (cid:54)= j.\n\nmaximize\n\nT(cid:88)\nT(cid:88)\n\nt=1\n\n+\n\n+\n\nt=1\nsubject to\n\ni=1\n\n(15)\nwhere \u03bbl,t,z \u2265 0 and \u03bbu,t,z \u2265 0 are the Lagrange multipliers for the band constraints. Then, we can\nthen state the following theoretical result (the proof easily follows from the proof of Theorem 2):\n\n\u221a\nTheorem 3 The expected regret of the optimal assignments for the problem de\ufb01ned by Eq. 15 with\nposterior samples for the thresholds is O(\n\nT ).\n\nRemark. In the above formulation, we do not enforce a speci\ufb01c mechanism to reduce disparate\nimpact\u2013our framework \ufb01nds the solution with maximum utility that satis\ufb01es the disparate impact\nconstraint. Depending on the distribution pY |X,Z and the de\ufb01nition of utility and bene\ufb01ts, such a\nsolution will result in an increase (decrease) of release rates for group z = 0 (z = 1) or viceversa.\n5 Experiments\nIn this section we empirically evaluate our framework on both synthetic and real data. To this end,\nwe compare the performance, in terms of both utility and fairness, of the following algorithms:\n\u2014 Optimal: Every decision is taken using the optimal decision rule d\u2217, which is de\ufb01ned by Eq. 3\nunder no fairness constraints and by Eq. 4 under fairness constraints.\n\u2014 Known: Every decision is taken by a judge following a (potentially biased) decision rule dv,\nas given by Eq. 5. The threshold for each judge is known and the assignment between judges and\ndecisions is found by solving the corresponding matching problem, i.e., Eq, 7 under no fairness\nconstraints and Eq. 12 under fairness constraints.\n\u2014 Unknown: Every decision is taken by a judge following a (potentially biased) decision rule dv,\nproceeding similarly as in \u201cKnown\". However, the threshold for each judge is unknown it is necessary\nto use posterior sampling to estimate the thresholds.\n\n6\n\n\f(a) Expected utility\n\n(b) Disparate impact\n\n(c) Regret\n\nFigure 1: Performance in synthetic data. Panels (a) and (b) show the trade-off between expected utility\nand disparate impact. For the utility, the higher the better and, for the disparate impact, the lower the\nbetter. Panel (c) shows the regret achieved by our algorithm under unknown experts\u2019 thresholds as\nde\ufb01ned in Eq. 10. Here, the solid lines show the results for m = 20 and dashed lines for m = 10.\n\u2014 Random: Every decision is taken by a judge following a (potentially biased) decision rule dv. The\nassignment between judges and decision is random.\n\n5.1 Experiments on Synthetic Data\nExperimental setup. For every decision, we \ufb01rst sample the sensitive attribute zi \u2208 {0, 1} from\nBernouilli(0.5) and then sample pY =1|xi,zi \u223c Beta(3, 5) if zi = 0 and from pY =1|xi,zi \u223c\nBeta(4, 3), otherwise. For every expert, we generate her decision thresholds \u03b8v,0 \u223c Beta(0.5, 0.5)\nand \u03b8v,1 \u223c Beta(5, 5). Here we assume there are n = 3m experts, to ensure that in each round there\nare at least m(1 \u2212 \u03b1) experts with \u03b8v,z < c, at least m(1 \u2212 \u03b1) experts with \u03b8v,z \u2265 c for z \u2208 {0, 1}.\nIn practice, if there is no feasible assignment for a round and desired level of fairness \u03b1, one may\ndecide to: i) add experts to the pool to increase diversity; ii) decrease the number of cases per round;\nor (iii) use a random assignment in that round. Finally, we set m = 20, T = 1000 and c = 0.5, and\nthe bene\ufb01cial outcome for an individual is d = 1, i.e., f (d) = d.\nResults. Figures 1(a)-(b) show the expected utility and the disparate impact after T units of time for\nthe optimal decision rule and for the group of experts under the assignments provided our algorithms\nand under random assignments. We \ufb01nd that the experts chosen by our algorithm provide decisions\nwith higher utility and lower disparate impact than the experts chosen at random, even if the thresholds\nare unknown. Moreover, if the threshold are known, the experts chosen by our algorithm closely\nmatch the performance of the optimal decision rule both in terms of utility and disparate impact.\nFinally, we compute the regret as de\ufb01ned by Eq. 10, i.e., the difference between the utilities provided\n\u221a\nby algorithm with Known and Unknown thresholds over time. Figure 1(c) summarizes the results,\nwhich show that, as time progresses, the regret degreases at a rate O(\n\nT ).\n\n5.2 Experiments on Real Data\nExperimental setup. We use the COMPAS recidivism prediction dataset compiled by ProPublica [8],\nwhich comprises of information about all criminal offenders screened through the COMPAS (Correc-\ntional Offender Management Pro\ufb01ling for Alternative Sanctions) tool in Broward County, Florida\nduring 2013-2014. In particular, for each offender, it contains a set of demographic features (gender,\nrace, age), the offender\u2019s criminal history (e.g., the reason why the person was arrested, number of\nprior offenses), and the risk score assigned to the offender by COMPAS. Moreover, ProPublica also\ncollected whether or not these individuals actually recidivated within two years after the screening.\nIn our experiments, the sensitive attribute z \u2208 {0, 1} is the race (white, black), the label y indicates\nwhether the individual recidivated (y = 1) or not (y = 0), the decision rule d speci\ufb01es whether\nan individual is released from jail (d = 0) or not (d = 1) and, for each sensitive attribute z, we\napproximate pY |X,Z=z using a logistic regression classi\ufb01er, which we train on 25% of the data.\nThen, we use the remaining 75% of the data to evaluate our algorithm as follows. Since we do not\nhave information about the identify of the judges who took each decision in the dataset, we create\nN = 3m (\ufb01ctitious) judges and sample their thresholds from a \u03b8 \u223c Beta(\u03c4, \u03c4 ), where \u03c4 controls the\ndiversity (lenient vs harsh) across judges by means the standard deviation of the distribution since\n4\u03c4 (2\u03c4 +1). Here, we consider two scenarios: (i) all experts are unbiased towards race and\nstd(\u03b8) =\nthus \u03b8v0 = \u03b8v1 and (ii) 50% of the experts are unbiased towards race and the other 50% are biased,\ni.e., \u03b8v1 = 1.2\u03b8v0. Finally, we consider m = 20 decisions per round, which results into 197 rounds,\nwhere we assign decisions to rounds at random.\nResults. Figure 2 shows the expected utility, the true utility and the disparate impact after T\nunits of time for the optimal decision rule and for the group of unbiased experts (scenario (i))\n\n1\n\n7\n\nUnfair= 0.2= 0.1= 0.05= 0.0100.10.20.3Disparate ImpactRandomOptimalKnown Unknown Unfair= 0.4= 0.2= 0.1= 0.0100.020.040.060.08UtilityUnfair= 0.4= 0.2= 0.1= 0.0100.050.10.150.20.25Disparate Impact0100200300400500T0.010.020.030.040.05RegretUnfair= 0.2= 0.1\f(a) Expected Utility\n\n(b) True Utility\n\n(c) Disparate Impact\n\nFigure 2: Performance in COMPAS data. Panels (a) and (b) show the expected utility and true utility\nand panel (c) shows the disparate impact. For the expected and true utility, the higher the better and,\nfor the disparate impact, the lower the better.\n\n(a) Known \u03b8\n\n(b) Unknown \u03b8\n\nt=1\n\n(cid:80)T\n\nFigure 3: Feasibility in COMPAS data. Probability that a round does not allow for an assignment\nbetween judges and decisions with less than \u03b1 disparate impact for different pools of experts of\nvarying diversity and percentage of biased judges.\n(cid:80)m\nunder the assignments provided our algorithms and under random assignments. The true utility\n\u02c6u\u2264T (d, c) is just the utility after T units of time given the actual true y values rather than pY |X,Z,\ni=1 d(xi(t), zi(t))(yi \u2212 c). Similarly as in the case of synthetic data,\ni.e., \u02c6u\u2264T (d, c) = 1\nT\nwe \ufb01nd that the judges chosen by our algorithm provide higher expected utility and true utility as\nwell as lower disparate impact than the judges chosen at random, even if the thresholds are unknown.\nWe notice that our algorithm achieves a level of fairness \u03b1 by decreasing the release rate of white\ndefendants, since our de\ufb01nition of utility penalizes more to release individuals who will be more likely\nto recidivate than to keep in jail those who will not. Note also that under fairness constraints, our\nalgorithm relies on a bi-criteria algorithm with a 1/2 approximation guarantee to solve the maximum\nweighted bipartite matching problem. As a consequence, it sometimes \ufb01nds matchings violate the\nfairness constraints but have higher utility.\nFigure 3 shows the probability that a round does not allow for an assignment between judges and\ndecisions with less than \u03b1 disparate impact for different pools of experts of varying diversity and\npercentage of biased judges. The results show that, on the one hand, our algorithms are able to ensure\nfairness more effectively if the pool of experts is diverse and, on the other hand, our algorithms are\nable to ensure fairness even if a signi\ufb01cant percentage of judges (e.g., 50%) are biased against a group\nof individuals sharing a certain sensitive attribute value.\n6 Conclusions\nIn this paper, we have proposed a set of practical algorithms to improve the utility and fairness of a\nsequential decision making process, where each decision is taken by a human expert, who is selected\nfrom a pool experts. Experiments on synthetic data and real jail-or-release decisions by judges show\nthat our algorithms are able to mitigate imperfect human decisions due to limited experience, implicit\nbiases or faulty probabilistic reasoning. Moreover, they also reveal that our algorithms bene\ufb01t from\nhigher diversity across the pool experts, being able to ensure fairness even if a signi\ufb01cant percentage\nof judges are biased against a group of individuals sharing a sensitive attribute value (e.g., race).\nThere are many interesting venues for future work. For example, in our work, we assumed all experts\nmake predictions using the same (true) conditional distribution and then apply (potentially) different\nthresholds. We have also assumed that experts do not learn from the decisions they take over time, i.e.,\ntheir prediction model and thresholds are \ufb01xed. It would be very interesting to relax these assumptions\nand account for experts with different prediction abilities. In some scenarios, a decision is taking\njointly by a group of experts, e.g., faculty recruiting decisions. It would be a natural follow-up to\nthe current work to design our algorithms for such scenario. Finally, in our experiments, we have to\ngenerate \ufb01ctitious judges since we do not have information about the identify of the judges who took\neach decision. It would be very valuable to gain access to datasets with such information [7].\n\n8\n\nUnfair= 0.2= 0.1= 0.05= 0.0100.10.20.3Disparate ImpactRandomOptimalKnown Unknown Unfair= 0.2= 0.1= 0.05= 0.0100.020.040.060.080.1Expected UtilityUnfair= 0.2= 0.1= 0.05= 0.0100.020.040.060.080.1True UtilityUnfair= 0.2= 0.1= 0.05= 0.0100.10.20.3Disparate Impact0.290.160.110.080.070.060.050.040.03Diversity, std()0.50.60.70.80.91Prob[round is feasible] = 0.1, bias = 0% = 0.1, bias = 50% = 0.01, bias = 0% = 0.01, bias = 50%0.290.160.110.080.070.060.050.040.03Diversity, std()0.50.60.70.80.91Prob[round is feasible] = 0.1, bias = 0% = 0.1, bias = 50% = 0.01, bias = 0% = 0.01, bias = 50%\fAcknowledgments. Isabel Valera acknowledges funding from a MPG Minerva Fast Track Grant.\nReferences\n[1] S. Barocas and A. D. Selbst. Big data\u00b4s disparate impact. California Law Review, 2016.\n\n[2] S. Boyd and L. Vandenberghe. Convex optimization. Cambridge university press, 2004.\n\n[3] S. Corbett-Davies, E. Pierson, A. Feller, S. Goel, and A. Huq. Algorithmic decision making\n\nand the cost of fairness. KDD, 2017.\n\n[4] C. Dwork, M. Hardt, T. Pitassi, O. Reingold, and R. Zemel. Fairness through awareness. In\n\nITCS, 2012.\n\n[5] M. Feldman, S. A. Friedler, J. Moeller, C. Scheidegger, and S. Venkatasubramanian. Certifying\n\nand removing disparate impact. In KDD, 2015.\n\n[6] M. Hardt, E. Price, N. Srebro, et al. Equality of opportunity in supervised learning. In NIPS,\n\n2016.\n\n[7] J. Kleinberg, H. Lakkaraju, J. Leskovec, J. Ludwig, and S. Mullainathan. Human decisions and\n\nmachine predictions. The Quarterly Journal of Economics, 133(1):237\u2013293, 2017.\n\n[8] J. Larson, S. Mattu, L. Kirchner, and J. Angwin. https://github.com/propublica/compas-analysis,\n\n2016.\n\n[9] M. Mastrolilli and G. Stamoulis. Constrained matching problems in bipartite graphs. In ISCO,\n\npages 344\u2013355. Springer, 2012.\n\n[10] C. Mu\u00f1oz, M. Smith, and D. Patil. Big Data: A Report on Algorithmic Systems, Opportunity,\n\nand Civil Rights. Executive Of\ufb01ce of the President. The White House., 2016.\n\n[11] I. Osband, D. Russo, and B. Van Roy. (more) ef\ufb01cient reinforcement learning via posterior\n\nsampling. In NIPS, pages 3003\u20133011, 2013.\n\n[12] D. B. West et al. Introduction to graph theory, volume 2. Prentice hall Upper Saddle River,\n\n2001.\n\n[13] B. Zafar, I. Valera, M. Gomez-Rodriguez, and K. Gummadi. Fairness beyond disparate treatment\n\n& disparate impact: Learning classi\ufb01cation without disparate mistreatment. In WWW, 2017.\n\n[14] B. Zafar, I. Valera, M. Gomez-Rodriguez, and K. Gummadi. Training fair classi\ufb01ers. AISTATS,\n\n2017.\n\n[15] B. Zafar, I. Valera, M. Gomez-Rodriguez, K. Gummadi, and A. Weller. From parity to\n\npreference: Learning with cost-effective notions of fairness. In NIPS, 2017.\n\n9\n\n\fA Proof sketch of Proposition 1\nConsider a simple setup with n = 2 experts and m = 1 decision at each round t \u2208 [T ]. Furthermore,\nwe \ufb01x the following two things before setting up the problem instance: (i) let g(\u00b7) be a deterministic\nfunction which computes a point estimate of a distribution (e.g., mean, or MAP); (ii) we assume a\ndeterministic tie-breaking by the assignment algorithm, and w.l.o.g. expert j = 1 is preferred over\nexpert j = 2 for assignment when both of them have same edge weights.\nFor the \ufb01rst expert j = 1, we know the exact value of the threshold \u03b81,z. For the second expert j = 2,\nthe threshold \u03b82,z could take any value in the range [0, 1] and we are given a prior distribution p(\u03b82,z).\n\nLet us denote(cid:101)\u03b82,z = g(cid:0)p(\u03b82,z)(cid:1). Now, we construct a problem instance for which the algorithm\nwould suffer linear regret separately for(cid:101)\u03b82,z > 0 and(cid:101)\u03b82,z = 0.\nProblem instance if(cid:101)\u03b82,z > 0\nWe consider a problem instance as follows: c = 0, \u03b82,z = 0, \u03b81,z = c+(cid:101)\u03b82,z\nhave pY =1|xi(t),zi(t) uniformly sampled from the range (c,(cid:101)\u03b82,z) (note that m = 1 and there is only\nj = 1 and has a cumulative expected utility of 3T(cid:101)\u03b82,z\nexpected utility of T(cid:101)\u03b82,z\nProblem instance if(cid:101)\u03b82,z = 0\nWe consider a problem instance as follows: c = 1, \u03b82,z = 1, \u03b81,z = c+(cid:101)\u03b82,z\nhave pY =1|xi(t),zi(t) uniformly sampled from the range ((cid:101)\u03b82,z, c) (note that m = 1 and there is only\nj = 1 and has a cumulative expected utility of \u2212T(cid:101)\u03b82,z\nexpected utility of 0. Hence, the algorithm suffers a linear regret of R(T ) = T(cid:101)\u03b82,z\n\none individual i = 1 at each round t). The algorithm would always assign the individual to expert\n. However, given the true thresholds, the\nalgorithm would have always assigned the individual to expert j = 2 and would have a cumulative\n\n, and for all t \u2208 [T ] we\n\n, and for all t \u2208 [T ] we\n\none individual i = 1 at each round t). The algorithm would always assign the individual to expert\n. However, given the true thresholds, the\nalgorithm would have always assigned the individual to expert j = 2 and would have a cumulative\n\n. Hence, the algorithm suffers a linear regret of R(T ) = T(cid:101)\u03b82,z\n\n.\n\n8\n\n2\n\n2\n\n2\n\n8\n\n.\n\n8\n\n8\n\n10\n\n\f", "award": [], "sourceid": 888, "authors": [{"given_name": "Isabel", "family_name": "Valera", "institution": "Max Planck Institute for Intelligent Systems"}, {"given_name": "Adish", "family_name": "Singla", "institution": "MPI-SWS"}, {"given_name": "Manuel", "family_name": "Gomez Rodriguez", "institution": "Max Planck Institute for Software Systems"}]}