{"title": "On Fairness and Calibration", "book": "Advances in Neural Information Processing Systems", "page_first": 5680, "page_last": 5689, "abstract": "The machine learning community has become increasingly concerned with the potential for bias and discrimination in predictive models. This has motivated a growing line of work on what it means for a classification procedure to be \"fair.\" In this paper, we investigate the tension between minimizing error disparity across different population groups while maintaining calibrated probability estimates. We show that calibration is compatible only with a single error constraint (i.e. equal false-negatives rates across groups), and show that any algorithm that satisfies this relaxation is no better than randomizing a percentage of predictions for an existing classifier. These unsettling findings, which extend and generalize existing results, are empirically confirmed on several datasets.", "full_text": "OnFairnessandCalibrationGeoffPleiss\u2217,ManishRaghavan\u2217,FelixWu,JonKleinberg,KilianQ.WeinbergerCornellUniversity,DepartmentofComputerScience{geoff,manish,kleinber}@cs.cornell.edu,{fw245,kwq4}@cornell.eduAbstractThemachinelearningcommunityhasbecomeincreasinglyconcernedwiththepotentialforbiasanddiscriminationinpredictivemodels.Thishasmotivatedagrowinglineofworkonwhatitmeansforaclassi\ufb01cationproceduretobe\u201cfair.\u201dInthispaper,weinvestigatethetensionbetweenminimizingerrordisparityacrossdifferentpopulationgroupswhilemaintainingcalibratedprobabilityestimates.Weshowthatcalibrationiscompatibleonlywithasingleerrorconstraint(i.e.equalfalse-negativesratesacrossgroups),andshowthatanyalgorithmthatsatis\ufb01esthisrelaxationisnobetterthanrandomizingapercentageofpredictionsforanexistingclassi\ufb01er.Theseunsettling\ufb01ndings,whichextendandgeneralizeexistingresults,areempiricallycon\ufb01rmedonseveraldatasets.1IntroductionRecently,therehasbeengrowingconcernabouterrorsofmachinelearningalgorithmsinsensitivedomains\u2013includingcriminaljustice,onlineadvertising,andmedicaltesting[33]\u2013whichmaysystematicallydiscriminateagainstparticulargroupsofpeople[2,4,8].Arecenthigh-pro\ufb01leexampleoftheseconcernswasraisedbythenewsorganizationProPublica,whostudiedarisk-assessmenttoolthatiswidelyusedinthecriminaljusticesystem.Thistoolassignstoeachcriminaldefendantanestimatedprobabilitythattheywillcommitafuturecrime.ProPublicafoundthattheriskestimatesassignedtodefendantswhodidnotcommitfuturecrimeswereonaveragehigheramongAfrican-AmericandefendantsthanCaucasiandefendants[1].Thisisaformoffalse-positiveerror,andinthiscaseitdisproportionatelyaffectedAfrican-Americandefendants.Tomitigateissuessuchasthese,themachinelearningcommunityhasproposeddifferentframeworksthatattempttoquantifyfairnessinclassi\ufb01cation[2,4,8,19,26,34,37].ArecentandparticularlynoteworthyframeworkisEqualizedOdds[19](alsoreferredtoasDisparateMistreatment[37]),1whichconstrainsclassi\ufb01cationalgorithmssuchthatnoerrortype(false-positiveorfalse-negative)disproportionatelyaffectsanypopulationsubgroup.Thisnotionofnon-discriminationisfeasibleinmanysettings,andresearchershavedevelopedtractablealgorithmsforachievingit[17,19,34,37].Whenrisktoolsareusedinpractice,akeygoalisthattheyarecalibrated:ifwelookatthesetofpeoplewhoreceiveapredictedprobabilityofp,wewouldlikeapfractionofthemembersofthissettobepositiveinstancesoftheclassi\ufb01cationproblem[11].Moreover,ifweareconcernedaboutfairnessbetweentwogroupsG1andG2(e.g.African-Americandefendantsandwhitedefendants)thenwewouldlikethiscalibrationconditiontoholdsimultaneouslyforthesetofpeoplewithineachofthesegroupsaswell[16].Calibrationisacrucialconditionforrisktoolsinmanysettings.Ifarisktoolforevaluatingdefendantswerenotcalibratedwithrespecttogroupsde\ufb01nedbyrace,forexample,thenaprobabilityestimateofpcouldcarrydifferentmeaningforAfrican-Americanandwhitedefendants,andhencethetoolwouldhavetheunintendedandhighlyundesirableconsequenceofincentivizingjudgestotakeraceintoaccountwheninterpretingitspredictions.Despitethe\u2217Equalcontribution,alphebeticalorder.1Fortheremainderofthepaper,wewilluseEqualizedOddstorefertothisnotionofnon-discrimination.31stConferenceonNeuralInformationProcessingSystems(NIPS2017),LongBeach,CA,USA.\fimportanceofcalibrationasaproperty,ourunderstandingofhowitinteractswithotherfairnesspropertiesislimited.Weknowfromrecentworkthat,exceptinthemostconstrainedcases,itisimpossibletoachievecalibrationwhilealsosatisfyingEqualizedOdds[8,26].However,wedonotknowhowbesttoachieverelaxationsoftheseguaranteesthatarefeasibleinpractice.Ourgoalistofurtherinvestigatetherelationshipbetweencalibrationanderrorrates.WeshowthateveniftheEqualizedOddsconditionsarerelaxedsubstantially\u2013requiringonlythatweightedsumsofthegrouperrorratesmatch\u2013itisstillproblematictoalsoenforcecalibration.Weprovidenecessaryandsuf\ufb01cientconditionsunderwhichthiscalibratedrelaxationisfeasible.Whenfeasible,ithasauniqueoptimalsolutionthatcanbeachievedthroughpost-processingofexistingclassi\ufb01ers.Moreover,weprovideasimplepost-processingalgorithmto\ufb01ndthissolution:withholdingpredic-tiveinformationforrandomlychoseninputstoachieveparityandpreservecalibration.However,thissimplepost-processingmethodisfundamentallyunsatisfactory:althoughthepost-processedpredictionsofourinformation-withholdingalgorithmare\u201cfair\u201dinexpectation,mostpractitionerswouldobjecttothefactthatanon-trivialportionoftheindividualpredictionsarewithheldasaresultofcointosses\u2013especiallyinsensitivesettingssuchashealthcareorcriminaljustice.Theoptimalityofthisalgorithmthushastroublingimplicationsandshowsthatcalibrationanderror-ratefairnessareinherentlyatodds(evenbeyondtheinitialresultsby[8]and[26]).Finally,weevaluatethesetheoretical\ufb01ndingsempirically,comparingcalibratednotionsofnon-discriminationagainstthe(uncalibrated)EqualizedOddsframeworkonseveraldatasets.Theseexperimentsfurthersupportourconclusionthatcalibrationanderror-rateconstraintsareinmostcasesmutuallyincompatiblegoals.Inpracticalsettings,itmaybeadvisabletochooseonlyoneofthesegoalsratherthanattemptingtoachievesomerelaxednotionofboth.2RelatedWorkCalibratedprobabilityestimatesareconsiderednecessaryforempiricalriskanalysistools[4,10,12,16].Inpracticalapplications,uncalibratedprobabilityestimatescanbemisleadinginthesensethattheenduseroftheseestimateshasanincentivetomistrust(andthereforepotentiallymisuse)them.Wenotehoweverthatcalibrationdoesnotremoveallpotentialformisuse,astheenduser\u2019sbiasesmightcauseherorhimtotreatestimatesdifferentlybasedongroupmembership.Thereareseveralpost-processingmethodsforproducingcalibratedoutputsfromclassi\ufb01cationalgorithms.Forexample,PlattScaling[31]passesoutputsthroughalearnedsigmoidfunction,transformingthemintocalibratedprobabilities.HistogramBinningandIsotonicRegression[35]learnageneralmonotonicfunctionfromoutputstoprobabilities.See[30]and[18]forempiricalcomparisonsofthesemethods.EqualizedOdds[19],alsoreferredtoasDisparateMistreatment[37],ensuresthatnoerrortypedisproportionatelyaffectsanyparticulargroup.Hardtetal.[19]provideapost-processingtechniquetoachievethisframework,whileZafaretal.[37]introduceoptimizationconstraintstoachievenon-discriminationattrainingtime.Recently,thisframeworkhasreceivedsigni\ufb01cantattentionfromthealgorithmicfairnesscommunity.Researchershavefoundthatitisincompatiblewithothernotionsoffairness[8,9,26].Additionally,Woodworthetal.[34]demonstratethat,undercertainassumptions,post-processingmethodsforachievingnon-discriminationmaybesuboptimal.Alternativefairnessframeworksexistandarecontinuouslyproposed.Wehighlightseveraloftheseworks,thoughbynomeansofferacomprehensivelist.(Morethoroughreviewscanbefoundin[2,4,32]).Ithasbeenshownthat,undermostframeworksoffairness,thereisatrade-offbetweenalgorithmicperformanceandnon-discrimination[4,9,19,39].SeveralworksapproachfairnessthroughthelensofStatisticalParity[6,7,14,20,22,23,29,38].Underthisde\ufb01nition,groupmembershipshouldnotaffectthepredictionofaclassi\ufb01er,i.e.membersofdifferentgroupsshouldhavethesameprobabilityofreceivingapositive-classprediction.However,ithasbeenarguedthatStatisticalParitymaynotbeapplicableinmanyscenarios[8,13,19,26],asitattemptstoguaranteeequalrepresentation.Forexample,itisinappropriateincriminaljustice,wherebaseratesdifferacrossdifferentgroups.ArelatednotionisDisparateImpact[15,36],whichstatesthatthepredictionratesforanytwogroupsshouldnotdifferbymorethan80%(anumbermotivatedbylegalrequirements).Dworketal.[13]introduceanotionoffairnessbasedontheideathatsimilarindividualsshouldreceivesimilaroutcomes,thoughitchallengingtoachievethisnotioninpractice.Fairnesshasalsobeenconsideredinonlinelearning[21,24],unsupervisedlearning[5],andcausalinference[25,27].2\f3ProblemSetupThesetupofourframeworkmostfollowstheEqualizedOddsframework[19,37];however,weextendtheirframeworkforusewithprobabilisticclassi\ufb01ers.LetP\u2282Rk\u00d7{0,1}betheinputspaceofabinaryclassi\ufb01cationtask.Inourcriminaljusticeexample,(x,y)\u223cPrepresentsaperson,withxrepresentingtheindividual\u2019shistoryandyrepresentingwhetherornotthepersonwillcommitanothercrime.Additionally,weassumethepresenceoftwogroupsG1,G2\u2282P,whichrepresentdisjointpopulationsubsets,suchasdifferentraces.Weassumethatthegroupshavedifferentbaserates\u00b5t,orprobabilitiesofbelongingtothepositiveclass:\u00b51=P(x,y)\u223cG1[y=1]6=P(x,y)\u223cG2[y=1]=\u00b52.Finally,leth1,h2:Rk\u2192[0,1]bebinaryclassi\ufb01ers,whereh1classi\ufb01essamplesfromG1andh2classi\ufb01essamplesfromG2.2Eachclassi\ufb01eroutputstheprobabilitythatagivensamplexbelongstothepositiveclass.ThenotionofEqualizedOddsnon-discriminationisbasedonthefalse-positiveandfalse-negativeratesforeachgroup,whichwegeneralizehereforusewithprobabilisticclassi\ufb01ers:De\ufb01nition1.Thegeneralizedfalse-positiverateofclassi\ufb01erhtforgroupGtiscfp(ht)=E(x,y)\u223cGt(cid:2)ht(x)|y=0(cid:3).Similarly,thegeneralizedfalse-negativerateofclassi\ufb01erhtiscfn(ht)=E(x,y)\u223cGt(cid:2)(1\u2212ht(x))|y=1(cid:3).Iftheclassi\ufb01erweretooutputeither0or1,thisrepresentsthestandardnotionsoffalse-positiveandfalse-negativerates.Wenowde\ufb01netheEqualizedOddsframework(generalizedforprobabilisticclassi\ufb01ers),whichaimstoensurethaterrorsofagiventypearenotbiasedagainstanygroup.De\ufb01nition2(ProbabilisticEqualizedOdds).Classi\ufb01ersh1andh2exhibitEqualizedOddsforgroupsG1andG2ifcfp(h1)=cfp(h2)andcfn(h1)=cfn(h2).CalibrationConstraints.Asstatedintheintroduction,thesetwoconditionsdonotnecessarilypreventdiscriminationiftheclassi\ufb01erpredictionsdonotrepresentwell-calibratedprobabilities.Recallthatcalibrationintuitivelysaysthatprobabilitiesshouldcarrysemanticmeaning:ifthereare100peopleinG1forwhomh1(x)=0.6,thenweexpect60ofthemtobelongtothepositiveclass.De\ufb01nition3.Aclassi\ufb01erhtisperfectlycalibratedif\u2200p\u2208[0,1],P(x,y)\u223cGt(cid:2)y=1|ht(x)=p(cid:3)=p.Itiscommonlyacceptedamongstpractitionersthatbothclassi\ufb01ersh1andh2shouldbecalibratedwithrespecttogroupsG1andG2topreventdiscrimination[4,10,12,16].Intuitively,thispreventstheprobabilityscoresfromcarryinggroup-speci\ufb01cinformation.Unfortunately,Kleinbergetal.[26](aswellas[8],inabinarysetting)provethataclassi\ufb01ercannotachievebothcalibrationandEqualizedOdds,eveninanapproximatesense,exceptinthemosttrivialofcases.3.1GeometricCharacterizationofConstraintsWenowwillcharacterizethecalibrationanderror-rateconstraintswithsimplegeometricintuitions.Throughouttherestofthispaper,allofourresultscanbeeasilyderivedfromthisinterpretation.Webeginbyde\ufb01ningtheregionofclassi\ufb01erswhicharetrivial,orthosethatoutputaconstantvalueforallinputs(i.e.hc(x)=c,where0\u2264c\u22641isaconstant).Wecanvisualizetheseclassi\ufb01ersonagraphwithgeneralizedfalse-positiveratesononeaxisandgeneralizedfalse-negativesontheother.Itfollowsfromthede\ufb01nitionsofgeneralizedfalse-positive/false-negativeratesandcalibrationthatalltrivialclassi\ufb01ershlieonthediagonalde\ufb01nedbycfp(h)+cfn(h)=1(Figure1a).Therefore,allclassi\ufb01ersthatare\u201cbetterthanrandom\u201dmustliebelowthisdiagonalinfalse-positive/false-negativespace(thegraytriangleinthe\ufb01gure).Anyclassi\ufb01erthatliesabovethediagonalperforms\u201cworsethanrandom,\u201daswecan\ufb01ndapointonthetrivialclassi\ufb01erdiagonalwithlowerfalse-positiveandfalse-negativerates.Nowwewillcharacterizethesetofcalibratedclassi\ufb01ersforgroupsG1andG2,whichwedenoteasH\u22171andH\u22172.Kleinbergetal.showthatthegeneralizedfalse-positiveandfalse-negativeratesofacalibratedclassi\ufb01erarelinearlyrelatedbythebaserateofthegroup:3cfn(ht)=(1\u2212\u00b5t)/\u00b5tcfp(ht).(1)2Inpractice,h1andh2canbetrainedjointly(i.e.theyarethesameclassi\ufb01er).3Throughoutthisworkwewilltreatthecalibrationconstraintasholdingexactly;however,ourresultsgeneralizetoapproximatesettingsaswell.SeetheSupplementaryMaterialsformoredetails.3\f11Generalized FP RateGeneralized FN RateH\u21e41H\u21e42\u00b52\u00b511\u00b521\u00b510h\u00b52h\u00b51(a)Possiblecal.classi-\ufb01ersH\u22171,H\u22172(blue/red).11Generalized FP RateGeneralized FN Rate0h2h1(b)Satisfyingcal.andequalF.P.rates.11Generalized FP RateGeneralized FN Rate0h2h1(c)Satisfyingcal.andequalF.N.rates.11Generalized FP RateGeneralized FN Rate0h2h1(d)Satisfyingcal.andageneralconstraint.Figure1:Calibration,trivialclassi\ufb01ers,andequal-costconstraints\u2013plottedinthefalse-pos./false-neg.plane.H\u22171,H\u22172arethesetofcal.classi\ufb01ersforthetwogroups,andh\u00b51,h\u00b52aretrivialclassi\ufb01ers.11Generalized FP RateGeneralized FN RateH\u21e41H\u21e420g=1g=2g=3(a)Level-ordercurvesofcost.Lowcostimplieslowerrorrates.11Generalized FP RateGeneralized FN Rate0h2h1h\u00b52\u02dch2(b)Usually,thereisacal-ibratedclassi\ufb01er\u02dch2withthesamecostofh1.11Generalized FP RateGeneralized FN Rate0h2h1(c)Cal.andequal-costareincompatibleifh1hashigherror.11Generalized FP RateGeneralized FN Rate0h2h\u00b52(d)Possiblecal.classi-\ufb01ersforG2(boldred)bymixingh2andh\u00b52.Figure2:Calibration-PreservingParitythroughinterpolation.Inotherwords,h1liesonalinewithslope(1\u2212\u00b51)/\u00b51andh2liesonalinewithslope(1\u2212\u00b52)/\u00b52(Figure1a).Thelowerendpointofeachlineistheperfectclassi\ufb01er,whichassignsthecorrectpredictionwithcompletecertaintytoeveryinput.Theupperendpointisatrivialclassi\ufb01er,asnocalibratedclassi\ufb01ercanperform\u201cworsethanrandom\u201d(seeLemma3inSectionS2).Theonlytrivialclassi\ufb01erthatsatis\ufb01esthecalibrationconditionforagroupGtistheonethatoutputsthebaserate\u00b5t.Wewillrefertoh\u00b51andh\u00b52asthetrivialclassi\ufb01ers,calibratedforgroupsG1andG2respectively.Itfollowsfromthede\ufb01nitionsthatcfp(h\u00b51)=\u00b51andcfn(h\u00b51)=1\u2212\u00b51,andlikewiseforh\u00b52.Finally,itisworthnotingthatforcalibratedclassi\ufb01ers,alowerfalse-positiveratenecessarilycorrespondstoalowerfalse-negativerateandvice-versa.Inotherwords,foragivenbaserate,a\u201cbetter\u201dcalibratedclassi\ufb01erliesclosertotheoriginonthelineofcalibratedclassi\ufb01ers.ImpossibilityofEqualizedOddswithCalibration.Withthisgeometricintuition,wecanprovideasimpli\ufb01edproofofthemainimpossibilityresultfrom[26]:Theorem(ImpossibilityResult[26]).Leth1andh2beclassi\ufb01ersforgroupsG1andG2with\u00b516=\u00b52.h1andh2satisfytheEqualizedOddsandcalibrationconditionsifandonlyifh1andh2areperfectpredictors.Intuitively,thethreeconditionsde\ufb01neasetofclassi\ufb01erswhichisoverconstrained.EqualizedOddsstipulatesthattheclassi\ufb01ersh1andh2mustlieonthesamecoordinateinthefalse-positive/false-negativeplane.Ash1mustlieonthebluelineofcalibratedclassi\ufb01ersforH\u22171andh2ontheredlineH\u22172theycanonlysatisfyEOattheuniqueintersectionpoint\u2014theorigin(andlocationoftheperfectclassi\ufb01er).Thisimpliesthatunlessthetwoclassi\ufb01ersachieveperfectaccuracy,wemustrelaxtheEqualizedOddsconditionsifwewanttomaintaincalibration.4RelaxingEqualizedOddstoPreserveCalibrationInthissection,weshowthatasubstantiallysimpli\ufb01ednotionofEqualizedOddsiscompatiblewithcalibration.Weintroduceageneralrelaxationthatseekstosatisfyasingleequal-costconstraintwhilemaintainingcalibrationforeachgroupGt.WebeginwiththeobservationthatEqualized4\fOddssetsconstraintstoequalizefalse-positivescfp(ht)andfalse-negativescfn(ht).Tocaptureandgeneralizethis,wede\ufb01neacostfunctiongttobealinearfunctionincfp(ht)andcfn(ht)witharbitrarydependenceonthegroup\u2019sbaserate\u00b5t.Moreformally,acostfunctionforgroupGtisgt(ht)=atcfp(ht)+btcfn(ht)(2)whereatandbtarenon-negativeconstantsthatarespeci\ufb01ctoeachgroup(andthusmaydependon\u00b5t):seeFigure1d.Wealsomaketheassumptionthatforany\u00b5t,atleastoneofatandbtisnonzero,meaninggt(ht)=0ifandonlyifcfp(ht)=cfn(ht)=0.4Thisclassofcostfunctionsencompassesavarietyofscenarios.Asanexample,imagineanapplicationinwhichtheequalfalse-positiveconditionisessentialbutnotthefalse-negativecondition.Suchascenariomayariseinourrecidivism-predictionexample,ifwerequirethatnon-repeatoffendersofanyracearenotdisproportionatelylabeledashighrisk.Ifweplotthesetofcalibratedclassi\ufb01ersH\u22171andH\u22172onthefalse-positive/false-negativeplane,wecanseethatensuringthefalse-positiveconditionrequires\ufb01ndingclassi\ufb01ersh1\u2208H\u22171andh2\u2208H\u22172thatfallonthesameverticalline(Figure1b).Conversely,ifweinsteadchoosetosatisfyonlythefalse-negativecondition,wewould\ufb01ndclassi\ufb01ersh1andh2thatfallonthesamehorizontal(Figure1c).Finally,ifbothfalse-positiveandfalse-negativeerrorsincuranegativecostontheindividual,wemaychoosetoequalizeaweightedcombinationoftheerrorrates[3,4,8],whichcanbegraphicallydescribedbytheclassi\ufb01erslyingonaconvexandnegatively-slopedlevelset(Figure1d).Withthesede\ufb01nitions,wecanformallyde\ufb01neourrelaxation:De\ufb01nition4(RelaxedEqualizedOddswithCalibration).Givenacostfunctiongtoftheformin(2),classi\ufb01ersh1andh2achieveRelaxedEqualizedOddswithCalibrationforgroupsG1andG2ifbothclassi\ufb01ersarecalibratedandsatisfytheconstraintg1(h1)=g2(h2).Itisworthnotingthat,forcalibratedclassi\ufb01ers,anincreaseincoststrictlycorrespondstoanincreaseinboththefalse-negativeandfalse-positiverate.Thiscanbeinterpretedgraphically,asthelevel-ordercostcurvesliefurtherawayfromtheoriginascostincreases(Figure2a).Inotherwords,thecostfunctioncanalwaysbeusedasaproxyforeithererrorrate.5Feasibility.ItiseasytoseethatDe\ufb01nition4isalwayssatis\ufb01able\u2013inFigures1b,1c,and1dweseethattherearemanysuchsolutionsthatwouldlieonagivenlevel-ordercostcurvewhilemaintainingcalibration,includingthecaseinwhichbothclassi\ufb01ersareperfect.Inpractice,however,notallclassi\ufb01ersareachievable.Fortherestofthepaper,wewillassumethatwehaveaccessto\u201coptimal\u201d(butpossiblydiscriminatory)calibratedclassi\ufb01ersh1andh2suchthat,duetowhateverlimitationsthereareonthepredictabilityofthetask,weareunableto\ufb01ndotherclassi\ufb01ersthathavelowercostwithrespecttogt.Weallowh1andh2tobelearnedinanyway,aslongastheyarecalibrated.Withoutlossofgenerality,fortheremainderofthepaper,wewillassumethatg1(h1)\u2265g2(h2).Sincebyassumptionwehavenowayto\ufb01ndaclassi\ufb01erforG1withlowercostthanh1,ourgoalisthereforeto\ufb01ndaclassi\ufb01er\u02dch2withcostequaltoh1.Thispairofclassi\ufb01erswouldrepresentthelowestcost(andthereforeoptimal)setofclassi\ufb01ersthatsatis\ufb01escalibrationandtheequalcostconstraint.Foragivenbaserate\u00b5tandvalueofthecostfunctiongt,acalibratedclassi\ufb01er\u2019spositioninthegeneralizedfalse-positive/false-negativeplaneisuniquelydetermined(Figure2a).Thisisbecauseeachlevel-ordercurveofthecostfunctiongthasnegativeslopeinthisplane,andeachlevelordercurveonlyintersectsagroup\u2019scalibratedclassi\ufb01erlineonce.Inotherwords,thereisauniquesolutioninthefalse-positive/false-negativeplaneforclassi\ufb01er\u02dch2(Figure2b).Considertherangeofvaluesthatgtcantake.Asnotedabove,gt(ht)\u22650,withequalityifandonlyifhtistheperfectclassi\ufb01er.Ontheotherhand,thetrivialclassi\ufb01er(again,whichoutputstheconstant\u00b5tforallinputs)isthecalibratedclassi\ufb01erthatachievesmaximumcostforanygt(seeLemma3inSectionS2).Asaresult,thecostofaclassi\ufb01erforgroupGtisbetween0andgt(h\u00b5t).Thisnaturallyleadstoacharacterizationoffeasibility:De\ufb01nition4canbeachievedifandonlyifh1incurslesscostthangroupG2\u2019strivialclassi\ufb01erh\u00b52;i.e.ifg1(h1)\u2264g2(h\u00b52).ThiscanbeseengraphicallyinFigure2c,inwhichthelevel-ordercurveforg1(h1)doesnotintersectthesetofcalibratedclassi\ufb01ersforG2.Since,byassumption,wecannot\ufb01ndacalibratedclassi\ufb01erforG1withstrictlysmallercostthanh1,thereisnofeasiblesolution.Ontheotherhand,ifh1incurslesscostthanh\u00b52,thenwewillshowfeasibilitybyconstructionwithasimplealgorithm.AnAlgorithm.WhileitmaybepossibletoencodetheconstraintsofDe\ufb01nition4intothetrainingprocedureofh1andh2,itisnotimmediatelyobvioushowtodoso.Evennaturallyprobabilistic4Bycalibration,wecannothaveoneofcfp(ht)=0orcfn(ht)=0withouttheother,seeFigure1a.5Thisholdsevenforapproximatelycalibratedclassi\ufb01ers\u2014seeSectionS3.5\falgorithms,suchaslogisticregression,canbecomeuncalibratedinthepresenceofoptimizationconstraints(asisthecasein[37]).Itisnotstraightforwardtoencodethecalibrationconstraintiftheprobabilitiesareassumedtobecontinuous,andpost-processingcalibrationmethods[31,35]wouldbreakequal-costconstraintsbymodifyingclassi\ufb01erscores.Therefore,welooktoachievethecalibratedEqualizedOddsrelaxationbypost-processingexistingcalibratedclassi\ufb01ers.Again,givenh1andh2withg1(h1)\u2265g2(h2),wewanttoarriveatacalibratedclassi\ufb01er\u02dch2forgroupG2suchthatg1(h1)=g2(\u02dch2).Recallthat,underourassumptions,thiswouldbethebestpossiblesolutionwithrespecttoclassi\ufb01ercost.WeshowthatthiscostconstraintcanbeachievedbywithholdingpredictiveinformationforarandomlychosensubsetofgroupG2.Inotherwords,ratherthanalwaysreturningh2(x)forallsamples,wewilloccasionallyreturnthegroup\u2019smeanprobability(i.e.theoutputofthetrivialclassi\ufb01erh\u00b52).InLemma4inSectionS2,weshowthatif\u02dch2(x)=(cid:26)h\u00b52(x)=\u00b52withprobability\u03b1h2(x)withprobability1\u2212\u03b1(3)thenthecostof\u02dch2isalinearinterpolationbetweenthecostsofh2andh\u00b52(Figure2d).Moreformally,wehavethatg2(\u02dch2)=(1\u2212\u03b1)g2(h2)+\u03b1g2(h\u00b52)),andthussetting\u03b1=g1(h1)\u2212g2(h2)g2(h\u00b52)\u2212g2(h2)ensuresthatg2(\u02dch2)=g1(h1)asdesired(Figure2b).Moreover,thisrandomizationpreservescalibration(seeSectionS4).Algorithm1summarizesthismethod.Algorithm1AchievingCalibrationandanEqual-CostConstraintviaInformationWithholdingInput:classi\ufb01ersh1andh2s.t.g2(h2)\u2264g1(h1)\u2264g2(h\u00b52),holdoutsetPvalid.\u2022Determinebaserate\u00b52ofG2(usingPvalid)toproducetrivialclassi\ufb01erh\u00b52.\u2022Construct\u02dch2usingwith\u03b1=g1(h1)\u2212g2(h2)g2(h\u00b52)\u2212g2(h2),where\u03b1istheinterpolationparameter.returnh1,\u02dch2\u2014whicharecalibratedandsatisfyg1(h1)=g2(\u02dch2).Implications.Inacertainsense,Algorithm1isan\u201coptimal\u201dmethodbecauseitarrivesattheuniquefalse-negative/false-positivesolutionfor\u02dch2,where\u02dch2iscalibratedandhascostequaltoh1.Therefore(byourassumptions)wecan\ufb01ndnobetterclassi\ufb01ersthatsatisfyDe\ufb01nition4.Thissimpleresulthasstrongconsequences,asthetradeoffstosatisfybothcalibrationandtheequal-costconstraintareoftenunsatisfactory\u2014bothintuitivelyandexperimentally(aswewillshowinSection5).We\ufb01ndtwoprimaryobjectionstothissolution.First,itequalizescostssimplybymakingaclassi\ufb01erstrictlyworseforoneofthegroups.Second,itachievesthiscostincreasebywithholdinginformationonarandomlychosenpopulationsubset,makingtheoutcomeinequitablewithinthegroup(asmeasuredbyastandardmeasureofinequalityliketheGinicoef\ufb01cient).Duetotheoptimalityofthealgorithm,theformeroftheseissuesisunavoidableinanysolutionthatsatis\ufb01esDe\ufb01nition4.Thelatter,however,isslightlymoresubtle,andbringsupthequestionofindividualfairness(whatguaranteeswewouldlikeanalgorithmtomakewithrespecttoeachindividual)andhowitinteractswithgroupfairness(population-levelguarantees).Whilethiscertainlyisanimportantissueforfuturework,inthisparticularsetting,evenifonecould\ufb01ndanotheralgorithmthatdistributestheburdenofadditionalcostmoreequitably,anyalgorithmwillmakeatleastasmanyfalse-positive/false-negativeerrorsasAlgorithm1,andthesemisclassi\ufb01cationswillalwaysbetragictotheindividualswhomtheyaffect.Theperformancelossacrosstheentiregroupisoftensigni\ufb01cantenoughtomakethiscombinationofconstraintssomewhatworryingtouseinpractice,regardlessofthealgorithm.ImpossibilityofSatisfyingMultipleEqual-CostConstraints.Itisnaturaltoarguetheremightbemultiplecostfunctionsthatwewouldliketoequalizeacrossgroups.However,satisfyingmorethanonedistinctequal-costconstraint(i.e.differentcurvesintheF.P./F.N.plane)isinfeasible.Theorem1(Generalizedimpossibilityresult).Leth1andh2becalibratedclassi\ufb01ersforG1andG2withequalcostwithrespecttogt.If\u00b516=\u00b52,andifh1andh2alsohaveequalcostwithrespecttoadifferentcostfunctiong0t,thenh1andh2mustbeperfectclassi\ufb01ers.(ProofinSectionS5).Notethatthisisageneralizationoftheimpossibilityresultof[26].Furthermore,weshowinTheorem9(inSectionS5)thatthisholdsinanapproximatesense:ifcalibrationandmultipledistinctequal-costconstraintsareapproximatelyachievedbysomeclassi\ufb01er,thenthatclassi\ufb01ermusthaveapproximatelyzerogeneralizedfalse-positiveandfalse-negativerates.6\f(a)IncomePrediction.0.00.20.40.20.40.6GeneralizedF.N.RateEqualOdds(Derived)H\u22171H\u22172h1h2heo1heo20.00.20.4Calib.+EqualF.N.H\u22171H\u22172h1h2\u02dch1\u02dch2GeneralizedF.P.Rate(b)HealthPrediction.0.20.40.60.20.40.6GeneralizedF.N.RateEqualOdds(Derived)H\u22171H\u22172h1h2heo1heo20.20.40.6Calib.+EqualCostH\u22171H\u22172h1h2\u02dch1\u02dch2GeneralizedF.P.Rate(c)RecidivismPrediction.0.250.500.750.20.40.60.8GeneralizedF.N.RateEqualOdds(Trained)H\u22171H\u22172h1h2heo1heo20.250.500.75EqualOdds(Derived)H\u22171H\u22172h1h2heo1heo20.250.500.75Calib.+EqualF.P.H\u22171H\u22172h1h2\u02dch1\u02dch2GeneralizedF.P.RateFigure3:GeneralizedF.P.andF.N.ratesfortwogroupsunderEqualizedOddsandthecalibratedrelaxation.Diamondsrepresentpost-processedclassi\ufb01ers.PointsontheEqualizedOdds(trained)graphrepresentclassi\ufb01ersachievedbymodifyingconstrainthyperparameters.5ExperimentsInlightofthese\ufb01ndings,ourgoalistounderstandtheimpactofimposingcalibrationandanequal-costconstraintonreal-worlddatasets.Wewillempiricallyshowthat,inmanycases,thiswillresultinperformancedegradation,whilesimultaneouslyincreasingothernotionsofdisparity.Weperformexperimentsonthreedatasets:anincome-prediction,ahealth-prediction,andacriminalrecidivismdataset.Foreachtask,wechooseacostfunctionwithinourframeworkthatisappropriateforthegivenscenario.Webeginwithtwocalibratedclassi\ufb01ersh1andh2forgroupsG1andG2.Weassumethattheseclassi\ufb01erscannotbesigni\ufb01cantlyimprovedwithoutmoretrainingdataorfeatures.Wethenderive\u02dch2toequalizethecostswhilemaintainingcalibration.Theoriginalclassi\ufb01ersaretrainedonaportionofthedata,andthenthenewclassi\ufb01ersarederivedusingaseparateholdoutset.Tocompareagainstthe(uncalibrated)EqualizedOddsframework,wederiveF.P./F.N.matchingclassi\ufb01ersusingthepost-processingmethodof[19](EO-Derived).Onthecriminalrecidivismdataset,weadditionallylearnclassi\ufb01ersthatdirectlyencodetheEqualizedOddsconstraints,usingthemethodsof[37](EO-Trained).(SeeSectionS6fordetailedtrainingandpost-processingprocedures.)WevisualizemodelerrorratesonthegeneralizedF.P.andF.N.plane.Additionally,weplotthecalibratedclassi\ufb01erlinesforG1andG2tovisualizemodelcalibration.IncomePrediction.TheAdultDatasetfromUCIMachineLearningRepository[28]contains14demographicandoccupationalfeaturesforvariouspeople,withthegoalofpredictingwhetheraperson\u2019sincomeisabove$50,000.Inthisscenario,weseektoachievepredictionswithequalizedcostacrossgenders(G1representswomenandG2representsmen).WemodelascenariowheretheprimaryconcernisensuringequalgeneralizedF.N.ratesacrossgenders,whichwould,forexample,helpjobrecruiterspreventgenderdiscriminationintheformofunderestimatedsalaries.Thus,wechooseourcostconstrainttorequireequalgeneralizedF.N.ratesacrossgroups.InFigure3a,weseethattheoriginalclassi\ufb01ersh1andh2approximatelylieonthelineofcalibratedclassi\ufb01ers.Intheleftplot(EO-Derived),weseethatitispossibleto(approximately)matchbotherrorratesoftheclassi\ufb01ersatthecostofheo1deviatingfromthesetofcalibratedclassi\ufb01ers.Intherightplot,weseethatitisfeasibletoequalizethegeneralizedF.N.rateswhilemaintainingcalibration.h1and\u02dch2lieonthesamelevel-ordercurveofgt(representedbythedashed-grayline),andsimultaneouslyremainonthe\u201cline\u201dofcalibratedclassi\ufb01ers.Itisworthnotingthatachievingeithernotionofnon-discriminationrequiressomecosttoatleastoneofthegroups.However,maintainingcalibrationfurtherincreasesthedifferenceinF.P.ratesbetweengroups.Insomesense,thecalibratedframeworktradesoffonenotionofdisparityforanotherwhilesimultaneouslyincreasingtheoverallerrorrates.7\fHealthPrediction.TheHeartDatasetfromtheUCIMachineLearningRepositorycontains14processedfeaturesfrom906adultsin4geographicallocations.Thegoalofthisdatasetistoaccuratelypredictwhetherornotanindividualhasaheartcondition.Inthisscenario,wewouldliketoreducedisparitybetweenmiddle-agedadults(G1)andseniors(G2).Inthisscenario,weconsiderF.P.andF.N.tobothbeundesirable.Afalsepredictionofaheartconditioncouldresultinunnecessarymedicalattention,whilefalsenegativesincurcostfromdelayedtreatment.Wethereforeutilizethefollowingcostfunctiongt(ht)=rfpht(x)(1\u2212y)+rfn(1\u2212ht(x))y,whichessentiallyassignsaweighttobothF.N.andF.P.predictions.Inourexperiments,wesetrfp=1andrfn=3.IntherightplotofFigure3b,wecanseethatthelevel-ordercurvesofthecostfunctionformacurvedlineinthegeneralizedF.P./F.N.plane.Becauseouroriginalclassi\ufb01erslieapproximatelyonthesamelevel-ordercurve,littlechangeisrequiredtoequalizethecostsofh1and\u02dch2whilemaintainingcalibration.Thisistheonlyexperimentinwhichthecalibratedframeworkincurslittleadditionalcost,andthereforecouldbeconsideredaviableoption.However,itisworthnotingthat,inthisexample,theequal-costconstraintdoesnotexplicitlymatcheitheroftheerrortypes,andthereforethetwogroupswillinexpectationexperiencedifferenttypesoferrors.IntheleftplotofFigure3b(EO-Derived),weseethatitisalternativelyfeasibletoexplicitlymatchboththeF.P.andF.N.rateswhilesacri\ufb01cingcalibration.CriminalRecidivismPrediction.Finally,weexaminetheframeworksinthecontextofourmotivat-ingexample:criminalrecidivism.Asmentionedintheintroduction,AfricanAmericans(G1)receiveadisproportionatenumberofF.P.predictionsascomparedwithCaucasians(G2)whenautomatedrisktoolsareusedinpractice.Therefore,weaimtoequalizethegeneralizedF.P.rate.Inthisexperiment,wemodifythepredictionsmadebytheCOMPAStool[12],arisk-assessmenttoolusedinpracticebytheAmericanlegalsystem.Additionally,wealsoseeifitispossibletoimprovetheclassi\ufb01erswithtraining-timeEqualizedOddsconstraintsusingthemethodsofZafaretal.[37](EO-Trained).InFigure3c,we\ufb01rstobservethattheoriginalclassi\ufb01ersh1andh2havelargegeneralizedF.P.andF.N.rates.BothmethodsofachievingEqualizedOdds\u2014trainingconstraints(leftplot)andpost-processing(middleplot)matchtheerrorrateswhilesacri\ufb01cingcalibration.However,weobservethat,assumingh1andh2cannotbeimproved,itisinfeasibletoachievethecalibratedrelaxation(Figure3cright).ThisisanexamplewherematchingtheF.P.rateofh1wouldrequireaclassi\ufb01erworsethanthetrivialclassi\ufb01erh\u00b52.Thisexamplethereforerepresentsaninstanceinwhichcalibrationiscompletelyincompatiblewithanyerror-rateconstraints.Iftheprimaryconcernofcriminaljusticepractitionersiscalibration[12,16],thentherewillinherentlybediscriminationintheformofF.P.andF.N.rates.However,iftheEqualizedOddsframeworkisadopted,themiscalibratedriskscoresinherentlycausediscriminationtoonegroup,asarguedintheintroduction.Therefore,themostmeaningfulchangeinsuchasettingwouldbeanimprovementtoh2(theclassi\ufb01erforAfricanAmericans)eitherthroughthecollectionofmoredataortheuseofmoresalientfeatures.Areductioninoverallerrortothegroupwithhighercostwillnaturallyleadtolesserror-ratedisparity.6DiscussionandConclusionWehaveobservedcasesinwhichcalibrationandrelaxedEqualizedOddsarecompatibleandcaseswheretheyarenot.Whenitisfeasible,thepenaltyofequalizingcostisampli\ufb01edifthebaseratesbetweengroupsdiffersigni\ufb01cantly.Thisisexpected,asbaseratedifferencesarewhatgiverisetocost-disparityinthecalibratedsetting.Seekingequalitywithrespecttoasingleerrorrate(e.g.false-negatives,asintheincomepredictionexperiment)willnecessarilyincreasedisparitywithrespecttotheothererror.Thismaybetolerable(intheincomepredictioncase,someemployeeswillendupover-paid)butcouldalsobehighlyproblematic(e.g.incriminaljusticesettings).Finally,wehaveobservedthatthecalibratedrelaxationisinfeasiblewhenthebest(discriminatory)classi\ufb01ersarenotfarfromthetrivialclassi\ufb01ers(leavinglittleroomforinterpolation).Insuchsettings,weseethatcalibrationiscompletelyincompatiblewithanequalizederrorconstraint.Insummary,weconcludethatmaintainingcostparityandcalibrationisdesirableyetoftendif\ufb01cultinpractice.Althoughweprovideanalgorithmtoeffectively\ufb01ndtheuniquefeasiblesolutiontobothconstraints,itisinherentlybasedonrandomlyexchangingthepredictionsofthebetterclassi\ufb01erwiththetrivialbaserate.Eveniffairnessisreachedinexpectation,foranindividualcase,itmaybehardtoacceptthatoccasionallyconsequentialdecisionsaremadebyrandomlywithholdingpredictiveinformation,irrespectiveofaparticularperson\u2019sfeaturerepresentation.Inthispaperwearguethat,aslongascalibrationisrequired,nolower-errorsolutioncanbeachieved.8\fAcknowledgementsGP,FW,andKQWaresupportedinpartbygrantsfromtheNationalScienceFoundation(III-1149882,III-1525919,III-1550179,III-1618134,andIII-1740822),theOf\ufb01ceofNavalResearchDOD(N00014-17-1-2175),andtheBillandMelindaGatesFoundation.MRissupportedbyanNSFGraduateResearchFellowship(DGE-1650441).JKissupportedinpartbyaSimonsInvestigatorAward,anAROMURIgrant,aGoogleResearchGrant,andaFacebookFacultyResearchGrant.References[1]J.Angwin,J.Larson,S.Mattu,andL.Kirchner.Machinebias:There\u2019ssoftwareusedacrossthecountrytopredictfuturecriminals.Andit\u2019sbiasedagainstblacks.ProPublica,2016.https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing.[2]S.BarocasandA.D.Selbst.Bigdata\u2019sdisparateimpact.CaliforniaLawReview,104,2016.[3]R.Berk.Aprimeronfairnessincriminaljusticeriskassessments.Criminology,41(6):6\u20139,2016.[4]R.Berk,H.Heidari,S.Jabbari,M.Kearns,andA.Roth.Fairnessincriminaljusticeriskassessments:Thestateoftheart.arXivpreprintarXiv:1703.09207,2017.[5]T.Bolukbasi,K.-W.Chang,J.Y.Zou,V.Saligrama,andA.T.Kalai.Manistocomputerprogrammeraswomanistohomemaker?debiasingwordembeddings.InNIPS,pages4349\u20134357,2016.[6]T.CaldersandS.Verwer.Threenaivebayesapproachesfordiscrimination-freeclassi\ufb01cation.KDD,2012.[7]T.Calders,F.Kamiran,andM.Pechenizkiy.Buildingclassi\ufb01erswithindependencyconstraints.InICDMWorkshops,2009.[8]A.Chouldechova.Fairpredictionwithdisparateimpact:Astudyofbiasinrecidivismpredictioninstruments.arXivpreprintarXiv:1703.00056,2017.[9]S.Corbett-Davies,E.Pierson,A.Feller,S.Goel,andA.Huq.Algorithmicdecisionmakingandthecostoffairness.InKDD,pages797\u2013806,2017.[10]C.S.Crowson,E.J.Atkinson,andT.M.Therneau.Assessingcalibrationofprognosticriskscores.StatisticalMethodsinMedicalResearch,25(4):1692\u20131706,2016.[11]A.P.Dawid.Thewell-calibratedbayesian.JournaloftheAmericanStatisticalAssociation,77(379):605\u2013610,1982.[12]W.Dieterich,C.Mendoza,andT.Brennan.COMPASriskscales:Demonstratingaccuracyequityandpredictiveparity.Technicalreport,Northpointe,July2016.http://www.northpointeinc.com/northpointe-analysis.[13]C.Dwork,M.Hardt,T.Pitassi,O.Reingold,andR.Zemel.Fairnessthroughawareness.InInnovationsinTheoreticalComputerScience,2012.[14]H.EdwardsandA.Storkey.Censoringrepresentationswithanadversary.InICLR,2016.[15]M.Feldman,S.A.Friedler,J.Moeller,C.Scheidegger,andS.Venkatasubramanian.Certifyingandremovingdisparateimpact.InKDD,pages259\u2013268,2015.[16]A.Flores,C.Lowenkamp,andK.Bechtel.Falsepositives,falsenegatives,andfalseanalyses:Arejoinderto\u201cmachinebias:There\u2019ssoftwareusedacrossthecountrytopredictfuturecriminals.andit\u2019sbiasedagainstblacks.\u201d.Technicalreport,Crime&JusticeInstitute,September2016.http://www.crj.org/cji/entry/false-positives-false-negatives-and-false-analyses-a-rejoinder.[17]G.Goh,A.Cotter,M.Gupta,andM.P.Friedlander.Satisfyingreal-worldgoalswithdatasetconstraints.InNIPS,pages2415\u20132423.2016.[18]C.Guo,G.Pleiss,Y.Sun,andK.Q.Weinberger.Oncalibrationofmodernneuralnetworks.InICML,2017.[19]M.Hardt,E.Price,andS.Nathan.Equalityofopportunityinsupervisedlearning.InAdvancesinNeuralInformationProcessingSystems,2016.9\f[20]J.E.JohndrowandK.Lum.Analgorithmforremovingsensitiveinformation:applicationtorace-independentrecidivismprediction.arXivpreprintarXiv:1703.04957,2017.[21]M.Joseph,M.Kearns,J.H.Morgenstern,andA.Roth.Fairnessinlearning:Classicandcontextualbandits.InNIPS,2016.[22]F.KamiranandT.Calders.Classifyingwithoutdiscriminating.InInternationalConferenceonComputerControlandCommunication,2009.[23]T.Kamishima,S.Akaho,andJ.Sakuma.Fairness-awarelearningthroughregularizationapproach.InICDMWorkshops,2011.[24]M.Kearns,A.Roth,andZ.S.Wu.Meritocraticfairnessforcross-populationselection.InInternationalConferenceonMachineLearning,pages1828\u20131836,2017.[25]N.Kilbertus,M.Rojas-Carulla,G.Parascandolo,M.Hardt,D.Janzing,andB.Sch\u00f6lkopf.Avoidingdiscriminationthroughcausalreasoning.InNIPS,2017.[26]J.Kleinberg,S.Mullainathan,andM.Raghavan.Inherenttrade-offsinthefairdeterminationofriskscores.InInnovationsinTheoreticalComputerScience.ACM,2017.[27]M.J.Kusner,J.R.Loftus,C.Russell,andR.Silva.Counterfactualfairness.arXivpreprintarXiv:1703.06856,2017.[28]M.Lichman.UCImachinelearningrepository,2013.URLhttp://archive.ics.uci.edu/ml.[29]C.Louizos,K.Swersky,Y.Li,M.Welling,andR.Zemel.Thevariationalfairautoencoder.InICLR,2016.[30]A.Niculescu-MizilandR.Caruana.Predictinggoodprobabilitieswithsupervisedlearning.InICML,2005.[31]J.Platt.Probabilisticoutputsforsupportvectormachinesandcomparisonstoregularizedlikelihoodmethods.AdvancesinLargeMarginClassi\ufb01ers,10(3):61\u201374,1999.[32]A.RomeiandS.Ruggieri.Amultidisciplinarysurveyondiscriminationanalysis.TheKnowledgeEngineeringReview,29(05):582\u2013638,2014.[33]White-House.Bigdata:Areportonalgorithmicsystems,opportunity,andcivilrights.Technicalreport,May2016.[34]B.Woodworth,S.Gunasekar,M.I.Ohannessian,andN.Srebro.Learningnon-discriminatorypredictors.InProceedingsofthe2017ConferenceonLearningTheory,volume65,pages1920\u20131953,Amsterdam,Netherlands,07\u201310Jul2017.PMLR.[35]B.ZadroznyandC.Elkan.Obtainingcalibratedprobabilityestimatesfromdecisiontreesandnaivebayesianclassi\ufb01ers.InICML,pages609\u2013616,2001.[36]M.B.Zafar,I.Valera,M.G.Rodriguez,andK.P.Gummadi.Learningfairclassi\ufb01ers.arXivpreprintarXiv:1507.05259,2015.[37]M.B.Zafar,I.Valera,M.G.Rodriguez,andK.P.Gummadi.Fairnessbeyonddisparatetreatment&disparateimpact:Learningclassi\ufb01cationwithoutdisparatemistreatment.InWorldWideWebConference,2017.[38]R.S.Zemel,Y.Wu,K.Swersky,T.Pitassi,andC.Dwork.Learningfairrepresentations.InICML,2013.[39]I.Zliobaite.Ontherelationbetweenaccuracyandfairnessinbinaryclassi\ufb01cation.InICMLWorkshoponFairness,Accountability,andTransparencyinMachineLearning,2015.10\f", "award": [], "sourceid": 2904, "authors": [{"given_name": "Geoff", "family_name": "Pleiss", "institution": "Cornell University"}, {"given_name": "Manish", "family_name": "Raghavan", "institution": "Cornell University"}, {"given_name": "Felix", "family_name": "Wu", "institution": null}, {"given_name": "Jon", "family_name": "Kleinberg", "institution": "Cornell University"}, {"given_name": "Kilian", "family_name": "Weinberger", "institution": "Cornell University"}]}