{"title": "Learning Fairness in Multi-Agent Systems", "book": "Advances in Neural Information Processing Systems", "page_first": 13854, "page_last": 13865, "abstract": "Fairness is essential for human society, contributing to stability and productivity. Similarly, fairness is also the key for many multi-agent systems. Taking fairness into multi-agent learning could help multi-agent systems become both efficient and stable. However, learning efficiency and fairness simultaneously is a complex, multi-objective, joint-policy optimization. To tackle these difficulties, we propose FEN, a novel hierarchical reinforcement learning model. We first decompose fairness for each agent and propose fair-efficient reward that each agent learns its own policy to optimize. To avoid multi-objective conflict, we design a hierarchy consisting of a controller and several sub-policies, where the controller maximizes the fair-efficient reward by switching among the sub-policies that provides diverse behaviors to interact with the environment. FEN can be trained in a fully decentralized way, making it easy to be deployed in real-world applications. Empirically, we show that FEN easily learns both fairness and efficiency and significantly outperforms baselines in a variety of multi-agent scenarios.", "full_text": "Learning Fairness in Multi-Agent Systems\n\nJiechuan Jiang\nPeking University\n\njiechuan.jiang@pku.edu.cn\n\nZongqing Lu\u2217\nPeking University\n\nzongqing.lu@pku.edu.cn\n\nAbstract\n\nFairness is essential for human society, contributing to stability and productivity.\nSimilarly, fairness is also the key for many multi-agent systems. Taking fairness\ninto multi-agent learning could help multi-agent systems become both ef\ufb01cient\nand stable. However, learning ef\ufb01ciency and fairness simultaneously is a complex,\nmulti-objective, joint-policy optimization. To tackle these dif\ufb01culties, we propose\nFEN, a novel hierarchical reinforcement learning model. We \ufb01rst decompose\nfairness for each agent and propose fair-ef\ufb01cient reward that each agent learns its\nown policy to optimize. To avoid multi-objective con\ufb02ict, we design a hierarchy\nconsisting of a controller and several sub-policies, where the controller maximizes\nthe fair-ef\ufb01cient reward by switching among the sub-policies that provides diverse\nbehaviors to interact with the environment. FEN can be trained in a fully decentral-\nized way, making it easy to be deployed in real-world applications. Empirically,\nwe show that FEN easily learns both fairness and ef\ufb01ciency and signi\ufb01cantly\noutperforms baselines in a variety of multi-agent scenarios.\n\n1\n\nIntroduction\n\nFairness is essential for human society, contributing to stability and productivity. Similarly, fairness\nis also the key for many multi-agent systems, e.g., routing [1], traf\ufb01c light control [2], and cloud\ncomputing [3]. More speci\ufb01cally, in routing, link bandwidth needs to be fairly allocated to packets to\nachieve load balance; in traf\ufb01c light control, resources provided by infrastructure needs to be fairly\nshared by vehicles; in cloud computing, resource allocation of virtual machines has to be fair to\noptimize pro\ufb01t.\nMany game-theoretic methods [4, 5, 6] have been proposed for fair division in multi-agent systems,\nwhich mainly focus on proportional fairness and envy-freeness. Most of them are in static settings,\nwhile some [7, 8, 9] consider the dynamic environment. Recently, multi-agent reinforcement learning\n(RL) has been successfully applied to multi-agent sequential decision-making, such as [10, 11, 12,\n13, 14, 15, 16]. However, most of them try to maximize the reward of each individual agent or the\nshared reward among agents, without taking fairness into account. Only a few methods consider\nfairness, but are handcrafted for different applications [17, 18, 19], which all require domain-speci\ufb01c\nknowledge and cannot be generalized. Some methods [20, 21, 22] aim to encourage cooperation in\nsocial dilemmas but cannot guarantee fairness.\nTaking fairness into multi-agent learning could help multi-agent systems become both ef\ufb01cient and\nstable. However, learning ef\ufb01ciency (system performance) and fairness simultaneously is a complex,\nmulti-objective, joint-policy optimization. To tackle these dif\ufb01culties, we propose a novel hierarchical\nRL model, FEN, to enable agents to easily and effectively learn both ef\ufb01ciency and fairness. First,\nwe decompose fairness for each agent and propose fair-ef\ufb01cient reward that each agent learns its own\npolicy to optimize it. We prove that agents achieve Pareto ef\ufb01ciency and fairness is guaranteed in\nin\ufb01nite-horizon sequential decision-making if all agents maximize their own fair-ef\ufb01cient reward\n\n\u2217Corresponding author\n\n33rd Conference on Neural Information Processing Systems (NeurIPS 2019), Vancouver, Canada.\n\n\fand learn the optimal policies. However, the con\ufb02icting nature between fairness and ef\ufb01ciency\nmakes it hard for a single policy to learn effectively. To overcome this, we then design a hierarchy,\nwhich consists a controller and several sub-policies. The controller maximizes the fair-ef\ufb01cient\nreward by switching among the sub-policies which directly interact with the environment. One of the\nsub-policies is designated to maximize the environmental reward, and other sub-policies are guided\nby information-theoretic reward to explore diverse possible behaviors for fairness. Additionally,\naverage consensus, which is included in the fair-ef\ufb01cient reward, coordinates the policies of agents in\nfully decentralized multi-agent learning. By saying fully decentralized, we emphasize that there is no\ncentralized controller, agents exchange information locally, and they learn and execute based on only\nlocal information.\nWe evaluate FEN in three classic scenarios, i.e., job scheduling, the Mathew effect, and manufacturing\nplant. It is empirically demonstrated that FEN obtains both fairness and ef\ufb01ciency and signi\ufb01cantly\noutperforms existing methods. By ablation studies, we con\ufb01rm the hierarchy indeed helps agents to\nlearn more easily. Bene\ufb01ted from distributed average consensus, FEN can learn and execute in a fully\ndecentralized way, making it easy to be deployed in real-world applications.\n\n2 Related Work\n\nFairness. There are many existing works on fair division in multi-agent systems. Most of them\nfocus on static settings [4, 5, 6], where the information of entire resources and agents are known and\n\ufb01xed, while some of them work on dynamic settings [7, 8, 9], where resource availability and agents\nare changing. For multi-agent sequential decision-making, a regularized maximin fairness policy is\nproposed [8] to maximize the worst performance of agents while considering the overall performance,\nand the policy is computed by linear programming or game-theoretic approach. However, none of\nthese works are learning approach. Some multi-agent RL methods [17, 18, 19] have been proposed\nand handcrafted for resource allocation in speci\ufb01c applications, such as resource allocation on multi-\ncore systems[17], sharing network resources among UAVs[18], and balancing various resources in\ncomplex logistics networks[19]. However, all these methods require domain-speci\ufb01c knowledge\nand thus cannot be generalized. Some methods [20, 21, 22] are proposed to improve cooperation\nin social dilemmas. In [20], the reward is shaped for two-player Stag Hunt games, which could\nbe agents\u2019 average reward in its multi-player version. In [20], one agent\u2019s reward is set to be the\nweighted average reward of two agents in two-player Stag Hunt games to induce prosociality. By\nextending the inequity aversion model, a shaped reward is designed in [21] to model agent\u2019s envy\nand guilt. A reward network is proposed in [22] to generate intrinsic reward for each agent, which\nis evolved based on the group\u2019s collective reward. Although cooperation in social dilemmas helps\nimprove agents\u2019 sum reward, it does not necessarily mean the fairness is guaranteed.\nThe Matthew effect, summarized as the rich get richer and the poor get poorer, can be witnessed in\nmany aspects of human society [23], as well as in multi-agent systems, such as preferential attachment\nin networking [24, 25] and mining process in blockchain systems [26]. The Matthew effect causes\ninequality in society and also performance bottleneck in multi-agent systems. Learning fairness could\navoid the Matthew effect and help systems become stable and ef\ufb01cient.\nMulti-Agent RL. Many multi-agent RL models have been recently proposed, such as [10, 11, 12, 13,\n14, 15, 16], but all of them only consider ef\ufb01ciency. CommNet [11] and ATOC [13] use continuous\ncommunication for multi-agent cooperation. Opponent modeling [27, 28] learns to reason about\nother agents\u2019 behaviors or minds for better cooperation or competition. MADDPG [10] is designed\nfor mixed cooperative-competitive environments. In these models, each agent only focuses on\noptimizing its own local reward. Thus, more capable agents will obtain more rewards and fairness\nis not considered. VDN [14], QMIX [15], and COMA [16] are designed for the scenario where all\nagents jointly maximize a shared reward. The shared reward is not directly related to fairness. Even\nif the shared reward is de\ufb01ned as the sum of local rewards of all agents, we can easily see that higher\nreward sum does not mean fairer.\nHierarchical RL. To solve more complex tasks with sparse rewards or long time horizons and to\nspeed up the learning process, hierarchical RL trains multiple levels of policies. The higher level\npolicies give goals or options to the lower level policies and only the lowest level applies actions to\nthe environment. So, the higher levels are able to plan over a longer time horizon or a more complex\ntask. Learning a decomposition of complex tasks into sub-goals are considered in [29, 30, 31], while\n\n2\n\n\flearning options are considered in [32, 33, 34]. However, none of these hierarchical RL models can\nbe directly applied to learning both fairness and ef\ufb01ciency in multi-agent systems.\n\n3 Methods\n\nWe propose Fair-Ef\ufb01cient Network, FEN, to enable agents to learn both ef\ufb01ciency and fairness\nin multi-agent systems. Unlike existing work, we decompose fairness for each agent and propose\nfair-ef\ufb01cient reward, and each agent learns its own policy to optimize it. However, optimizing the\ntwo con\ufb02icting objectives is hard for a single learning policy. To this end, we propose a hierarchy\nspeci\ufb01cally designed for easing this learning dif\ufb01culty. The hierarchy consists a controller and several\nsub-policies, where the controller learns to select sub-policies and each sub-policy learns to interact\nwith the environment in a different way. Average consensus, which is included in the fair-ef\ufb01cient\nreward, coordinates agents\u2019 policies and enables agents to learn in a fully decentralized way.\n\n3.1 Fair-Ef\ufb01cient Reward\n\nn\u22121\n\ni=1\n\n\u00afu2\n\n(ui\u2212\u00afu)2\n\n(cid:80)n\n\n(cid:113) 1\n\nIn the multi-agent system we consider, there are n agents and limited resources in the environment.\nThe resources are non-excludable and rivalrous (common resources), e.g., CPU, memory, and network\nbandwidth. At each timestep, the environmental reward r an agent obtains is only related to its\noccupied resources at that timestep. We de\ufb01ne the utility of agent i at timestep t as ui\nj,\nj=0 ri\nwhich is the average reward over elapsed timesteps. We use the coef\ufb01cient of variation (CV) of agents\u2019\nutilities\nto measure fairness [35], where \u00afu is average utility of all agents. A\nsystem is said to be fairer if and only if the CV is smaller.\nIn multi-agent sequential decision-making, it is dif\ufb01cult for an individual agent to optimize the CV\nsince it is not just related to the agent\u2019s own policy, but the joint policies of all agents. However, as\nthe resources are limited, the upper bound of \u00afu can be easily reached by self-interested agents. Thus,\n\u00afu is hardly affected by an individual agent and the contribution of agent i to the variance could be\napproximated as (ui \u2212 \u00afu)2/\u00afu2. We decompose the fairness objective for each agent and propose the\nfair-ef\ufb01cient reward\n\n(cid:80)t\n\nt = 1\nt\n\nThe fair-ef\ufb01cient reward allows each agent to respond to the behaviors of other agents, which can be\nsummarized by \u00afu. Therefore, \u00afu can actually coordinate agents\u2019 policies in decentralized multi-agent\nlearning.\nProposition 1. The optimal fair-ef\ufb01cient policy set \u03c0\u2217 is Pareto ef\ufb01cient in in\ufb01nite-horizon sequential\ndecision-making.\nProof. We prove by contradiction. We \ufb01rst prove the resources must be fully occupied. Since\nthe decision-making is in\ufb01nite-horizon, the resources could be allocated in any proportion in the\ntime domain. Assume the resources are not fully used, there must exist another \u03c0(cid:48), under which\neach agent could occupy the remaining resources according to the ratio of ui/n\u00afu. Then, we have\n|ui(cid:48)\ni > Fi, which contradicts the\npre-condition that \u03c0\u2217 is optimal. We then prove \u03c0\u2217 is Pareto ef\ufb01cient. Assume Pareto ef\ufb01ciency is\nnot achieved, there must exist \u2200i, ui(cid:48) (cid:62) ui \u2227\u2203i, ui(cid:48)\ni=1 ui, which contradicts\nthe pre-condition that the resources are fully occupied.\nProposition 2. The optimal fair-ef\ufb01cient policy set \u03c0\u2217 achieves equal allocation when the resources\nare fully occupied.\nProof. We prove by contradiction. Assume the allocation is not equal when the resources are fully\noccupied, \u2203i, ui (cid:54)= \u00afu. There must exist another \u03c0(cid:48), under which agents that have ui > \u00afu can give up\n\n/\u00afu(cid:48) \u2212 1| = |ui/\u00afu \u2212 1|, but \u00afu(cid:48)/c > \u00afu/c. Thus, for each agent i, F (cid:48)\ni=1 ui(cid:48)\n\n> ui, so(cid:80)n\n\n>(cid:80)n\n\n3\n\n\u00afut/c\n\n\u02c6ri\nt =\n\n\u0001 +(cid:12)(cid:12)ui\n\nt/\u00afut \u2212 1(cid:12)(cid:12) ,\nof the system, encouraging the agent to improve ef\ufb01ciency;(cid:12)(cid:12)ui\nlearns its own policy to maximize the objective Fi = E(cid:2)(cid:80)\u221e\n\nwhere c is a constant that normalizes the numerator and is set to the maximum environmental reward\nthe agent obtains at a timestep. In the fair-ef\ufb01cient reward, \u00afut/c can be seen as the resource utilization\n\ndeviation from the average, and the agent will be punished no matter it is above or below the average,\nwhich leads to low variance; \u0001 is a small positive number to avoid zero denominator. Each agent i\n\nt/\u00afut \u2212 1(cid:12)(cid:12) measures the agent\u2019s utility\n(cid:3), where \u03b3 is the discount factor.\n\nt=0 \u03b3t\u02c6ri\nt\n\n\fresources to make ui = \u00afu. Then, for the remaining resources and other agents, this is an isomorphic\nsubproblem. According to Proposition 1, the resources will be fully occupied by other agents. After\nthat, we have F (cid:48)\ni > Fi, which contradicts the\npre-condition that \u03c0\u2217 is optimal.\n\ni > Fi. This process can be repeated until \u2200i, F (cid:48)\n\n3.2 Hierarchy\n\nA learning policy could feel ambiguous while considering both fairness and ef\ufb01ciency since they\nmight con\ufb02ict in some states. For example, if different behaviors of other agents cause the change of\n\u00afu, an agent may need to perform different action at a same state to maximize its fair-ef\ufb01cient reward.\nHowever, this is hard for a single learned policy.\nTo overcome this dif\ufb01culty, we design a hierarchy that\nconsists of a controller and several sub-policies param-\neterized respectively by \u03b8 and \u03c6 , illustrated in Figure 1.\nThe controller selects one of sub-policies by the sampled\nindex zt \u223c \u03c0\u03b8(\u00b7|ot) based on the partial observation ot.\nThe controller receives the fair-ef\ufb01cient reward \u02c6r and\nacts at a lower temporal resolution than the sub-policies.\nEvery T timesteps, the controller chooses a sub-policy\nand in the next T timesteps, the chosen sub-policy out-\nputs actions to interact with the environment.\nTo obtain ef\ufb01ciency, we designate one of the sub-policies\nparameterized by \u03c61 to maximize the reward r given by\nthe environment. For other sub-policies, we exploit an\ninformation-theoretic objective to guide the sub-policies\nto explore diverse possible behaviors for fairness.\nFrom the perspective of the controller, these sub-policies\nshould be able to be distinguished from each other and\nthus the controller can have more choices. Obviously, we\ncan not quantify the difference of sub-policies directly.\nHowever, the experienced observations under a sub-policy could indirectly re\ufb02ect the policy. The\nmore differences between sub-policies, the less the uncertainty of z is, given observation o under the\npolicy. That is to say, the mutual information I(Z; O) should be maximized by the sub-policy and\nwe take it as one term of the objective. On the other hand, to explore diverse possible behaviors, the\nsub-policy should act as randomly as possible, so we also maximize the entropy between the action\nand observation H(A|O). In summary, the objective of the sub-policy is to maximize\n\nFigure 1: FEN architecture.\n\nJ(\u03c6) = I(Z; O) + H(A|O)\n\n= H(Z) \u2212 H(Z|O) + H(A|O)\n\u2248 \u2212H(Z|O) + H(A|O)\n= Ez\u223c\u03c0\u03b8,o\u223c\u03c0\u03c6[log p(z|o)] + Ea\u223c\u03c0\u03c6[\u2212p(a|o) log p(a|o)].\n\nAs H(Z) is only related to \u03b8, H(Z) can be seen a constant and be neglected. The controller just\noutputs the probability p\u03b8(z|o), and thus the \ufb01rst term of the objective can be interpreted as that each\nsub-policy tries to maximize the expected probability that it would be selected by the controller. To\nmaximize it, we can give sub-policies a reward \u02dcr = log p\u03b8(z|o) at each timestep and use RL to train\nthe sub-policies. The second term can be treated as an entropy regularization, which is differentiable\nand could be optimized by backpropagation.\nThe hierarchy reduces the dif\ufb01culty of learning both ef\ufb01ciency and fairness. The controller focuses\non the fair-ef\ufb01cient reward and learns to decide when to optimize ef\ufb01ciency or fairness by selecting\nthe sub-policy, without directly interacting with the environment. Sub-policy \u03c61 learns to optimize\nthe environmental reward, i.e., ef\ufb01ciency. Other sub-policies learn diverse behaviors to meet the\ncontroller\u2019s demand of fairness. The fair-ef\ufb01cient reward changes slowly since it is slightly affected\nby immediate environmental reward the sub-policy obtains. Thus, the controller can plan over a long\ntime horizon to optimize both ef\ufb01ciency and fairness, while the sub-policies only optimize their own\nobjectives within the given time interval T .\n\n4\n\ncontroller....sub-policies(Gossip)\f3.3 Decentralized Training\n\nThe centralized policy has an advantage in coordinating all agents\u2019 behaviors. However, the central-\nized policy is hard to train, facing the curse of dimensionality as the number of agents increases. FEN\nis a decentralized policy in both training and execution. Although each agent only focuses on its own\nfair-ef\ufb01cient reward, they are coordinated by the average consensus on utility.\nIn the decentralized training, each agent need to perceive the average utility \u00afu to know its current\nutility deviation from the average. When the number of agents is small, it is easy to collect the utility\nfrom each agent and compute the average. When the number of agents is large, it may be costly to\ncollect the utility from each agent in real-world applications. To deal with this, we adopt a gossip\nalgorithm for distributed average consensus [36]. Each agent i maintains the average utility \u00afui and\niteratively updates it by\n\n\u00afui(t + 1) = \u00afui(t) +\n\nwij \u00d7 (\u00afuj(t) \u2212 \u00afui(t)),\n\n(cid:88)\n\nj\u2208Ni\n\nwhere \u00afui(0) = ui, Ni is the set of neighboring agents in agent i\u2019s observation, and the weight\nwij = 1/(max{di, dj} + 1), where the degree di = |Ni|. The gossip algorithm is distributed and\nrequires only limited communication between neighbors to estimate the average.\nThe training of FEN is detailed in Algo-\nrithm 1. The controller and sub-policies\nare trained both using PPO [37]. The con-\ntroller selects one sub-policy every T to in-\nteract with the environment. The selected\nsub-policy is updated based on the trajec-\ntory during T . The controller is updated\nbased on the trajectory of every sub-policy\nselection and its obtained fair-ef\ufb01cient re-\nward during each episode.\n\nAlgorithm 1 FEN training\n1: Initialize ui, \u00afui, the controller \u03b8 and sub-policies \u03c6\n2: for episode = 1, . . . ,M do\n3:\n4:\n5:\n\nThe chosen sub-policy \u03c6z acts to the environment\nand gets the reward\nif t%T = 0 then\n\nThe controller chooses one sub-policy \u03c6z\nfor t = 1, . . . , max-episode-length do\n\nlog p\u03b8(z|ot)\n\nif z = 1,\n\n(cid:26)\n\nelse\n\nrt\n\n4 Experiments\n\nUpdate \u03c6z using PPO\nUpdate \u00afui (with gossip algorithm)\nCalculate \u02c6ri =\nThe controller reselects one sub-policy\n\n\u0001+|ui/\u00afui\u22121|\n\n\u00afui/c\n\n6:\n7:\n8:\n9:\n10:\n11:\n12:\n13:\n14: end for\n\nFor the experiments, we design three sce-\nnarios as abstractions of job scheduling,\nthe Matthew effect, and manufacturing\nplant, which are illustrated in Figure 2.\nWe demonstrate that by each agent decen-\ntralizedly optimizing the fair-ef\ufb01cient reward the multi-agent system could obtain a great balance\nbetween ef\ufb01ciency and fairness, and that the hierarchy indeed helps to learn both fairness and ef-\n\ufb01ciency more easily. In the experiments, we compare FEN against several baselines which have\ndifferent optimization objectives and are summarized as follows.\n\nend if\nend for\nUpdate \u03b8 using PPO\n\n\u2022 Independent agents are fully self-interested and each agent maximizes its expected sum\n\u2022 Inequity Aversion agents receive a shaped reward ri \u2212 \u03b1\nN\u22121\n\nof discounted environmental rewards \u03c8i = E(cid:2)(cid:80)\u221e\n\n(cid:3).\n(cid:80) max(ri \u2212 rj, 0) to model the envy and guilt [21].\n\n(cid:80) max(rj \u2212 ri, 0) \u2212\n\nt=0 \u03b3tri\nt\n\n\u03b2\n\nN\u22121\n\nFigure 2: Illustration of experimental scenarios: job scheduling (left), the Matthew effect (mid),\nmanufacturing plant (right).\n\n5\n\nAgentResourcePac-manGhostAgentGems\f(cid:80) \u03c8i/n [20].\n\n\u2022 Avg agents take the average reward of agents as a shared reward and maximize avg\u03c8 =\n\n\u2022 Min agents consider the worst performance of agents and maximize min\u03c8.\n\u2022 Min+\u03b1Avg agents consider both the worst performance and system performance and maxi-\n\nmize the regularized maximin fairness [8], min\u03c8 + \u03b1avg\u03c8.\n\nNote that in the last three baselines, all agents share the same optimization objective. To ensure the\ncomparison is fair, the basic hyperparameters are all the same for FEN and the baselines, which\nare summarized in Appendix. The details about the experimental setting of each scenario are also\navailable in Appendix. Moreover, the code of FEN is at https://github.com/PKU-AI-Edge/FEN.\n\n4.1\n\nJob Scheduling\n\nIn this scenario of job scheduling, we investigate whether agents can learn to fairly and ef\ufb01ciently\nshare a resource. In a 5 \u00d7 5 grid world, there are 4 agents and 1 resource, illustrated in Figure 2 (left).\nThe resource\u2019s location is randomly initialized in different episodes, but \ufb01xed during an episode.\nEach agent has a local observation that contains a square view with 3 \u00d7 3 grids centered at the agent\nitself. At each timestep, each agent can move to one of four neighboring grids or stay at current grid.\nIf the agent occupies the resource (move to or stay at the resource\u2019s location), it receives a reward of\n1, which could be seen as the job is scheduled, otherwise the reward is 0. Two agents cannot stay at a\nsame grid, making sure the resource can only be occupied by one agent at a timestep. We trained all\nthe methods for \ufb01ve runs with different random seeds. All experimental results are presented with\nstandard deviation (also in other two scenarios). Moreover, as all agents are homogeneous in the task,\nwe let agents share weights for all the methods.\nTable 1 shows the performance of FEN and the baselines in terms of resource utilization (sum of\nutility), coef\ufb01cient of variation (CV) of utility (fairness), min utility, and max utility. Independent\nhas the highest resource utilization, but also the worst CV. Self-interested Independent agents\nwould not give up the resource for fairness, which is also witnessed by that min utility is 0 and max\nutility is 0.88, close to resource utilization. FEN has the lowest CV and the highest min utility. As\nonly one agent can use the resource at a time, fairly sharing the resource among agents inevitably\nincurs the reduction of resource utilization. However, FEN can obtain much better fairness at a\nsubtle cost of resource utilization, and its resource utilization is slightly less than Independent.\nMaximizing avg\u03c8 causes high CV since the average is not directly related to fairness. Its resource\nutilization is also lower, because avg\u03c8 is determined by all the agents, making it hard for individual\nagents to optimize by decentralized training. For the same reason, min\u03c8 is hard to be maximized by\nindividual agents. The regularized maximin fairness reward min\u03c8 + \u03b1avg\u03c8 is designed to obtain\na balance between fairness and resource utilization. However, due to the limitations of these two\nobjective terms, Min+\u03b1Avg is much worse than FEN. The CV of Inequity Aversion is better than\nIndependent but still worse than FEN, and the resource utilization is much lower, showing modeling\nenvy and guilt is not effective in fairness problems. Moreover, the hyperparameters \u03b1 and \u03b2 might\ngreatly affect the performance.\n\nTable 1: Job scheduling\nresource utilization\n\nCV\n\nIndependent\n\nInequity Aversion\n\nMin\nAvg\n\nFEN\n\nMin+\u03b1Avg\n\nFEN w/o Hierarchy\nMin\nAvg\n\ncentralized policy\n\nMin+\u03b1Avg\n\nw/ Hierarchy\n\nMin\nAvg\n\nMin+\u03b1Avg\n\n96% \u00b111%\n72% \u00b19%\n47% \u00b18%\n84% \u00b17%\n63% \u00b15%\n90% \u00b15%\n57% \u00b113%\n12% \u00b14%\n61% \u00b15%\n19% \u00b15%\n62% \u00b19%\n84% \u00b16%\n71% \u00b18%\n\n6\n\n0\n\n1.57 \u00b10.26\n0.04 \u00b10.01\n0.69 \u00b10.17\n0.07 \u00b10.02\n0.30 \u00b10.07\n0.05 \u00b10.03\n0.75 \u00b10.13\n0.39 \u00b10.03\n0.09 \u00b10.03\n0.17 \u00b10.05 0.18\u00b10.03\n0.22 \u00b10.06\n0.10 \u00b10.03\n0.82 \u00b10.11\n1.46 \u00b10.14\n0.57 \u00b10.05\n0.31 \u00b10.11\n0.61 \u00b10.14\n0.28 \u00b10.09\n\nmin utility max utility\n0.88 \u00b10.17\n0.35 \u00b10.12\n0.16 \u00b10.05\n0.46 \u00b10.17\n0.24 \u00b10.06\n0.28 \u00b10.07\n0.18 \u00b10.11\n0.06 \u00b10.03\n0.53 \u00b10.06\n0.09 \u00b10.03\n0.21 \u00b10.05\n0.41 \u00b10.07\n0.26 \u00b10.06\n\n0.02 \u00b10.01\n0.09 \u00b10.02\n0.08 \u00b10.03\n0.11 \u00b10.04\n\n0\n0\n\n\fFigure 3: Learning curves of FEN and FEN\nw/o Hierarchy in job scheduling.\n\nFigure 4: Probability of selecting different\nsub-policies in terms of (ui \u2212 \u00afu)/\u00afu.\n\nSince min\u03c8, avg\u03c8, and min\u03c8 + \u03b1avg\u03c8 do not depend on individual agents, but all agents, we adopt\na centralized policy, which takes all observations and outputs actions for all agents, to optimize each\nobjective. As shown in Table 1, the centralized policies for Min, Avg, and Min+\u03b1Avg are even worse\nthan their decentralized versions. Although the centralized policy could coordinate agents\u2019 behaviors,\nit is hard to train because of the curse of dimensionality. We also tried the centralized policy in the\nMatthew effect and manufacturing plant, but it did not work and thus is omitted.\nDoes the hierarchy indeed help the learning of FEN? To verify the effect of the hierarchy, we trained a\nsingle policy to maximize the fair-ef\ufb01cient reward directly without the hierarchy. Figure 3 illustrates\nthe learning curves of FEN and FEN w/o Hierarchy, where we can see that FEN converges to a\nmuch higher mean fair-ef\ufb01cient reward than FEN w/o Hierarchy. As shown in Table 1, although\nFEN w/o Hierarchy is fairer than other baselines, the resource utilization is mediocre. This is\nbecause it is hard for a single policy to learn ef\ufb01ciency from the fair-ef\ufb01cient reward. However, in\nFEN, one of the sub-policies explicitly optimizes the environmental reward to improve the ef\ufb01ciency,\nother sub-policies learn diverse fairness behaviors, and the controller optimizes fair-ef\ufb01cient reward\nby long time horizon planing. The hierarchy successfully decomposes the complex objective and\nreduce the learning dif\ufb01culty.\nTo further verify the effectiveness of the hierarchy, we use the hierarchy with other baselines. The\ncontroller maximizes each own objective and the sub-policies are the same as FEN. Table 1 shows\ntheir performance has a certain degree of improvement, especially the resource utilizations of Min\nand Min+\u03b1Avg raise greatly and the CV of Min+\u03b1Avg reduces signi\ufb01cantly. That demonstrates the\nhierarchy we proposed could reduce learning dif\ufb01culty in many general cases with both global and\nlocal objectives. However, these baselines with the hierarchy are still worse than FEN in both resource\nutilization and CV, verifying the effectiveness of the fair-ef\ufb01cient reward.\nIn order to analyze the behavior of the controller, in Figure 4 we visualize the probability of selecting\nsub-policy \u03c61 and other sub-policies in terms of the utility deviation from average, (ui \u2212 \u00afu)/\u00afu. It\nshows when the agent\u2019s utility is below average, the controller is more likely to select \u03c61 to occupy\nthe resources, and when the agent\u2019s utility is above average, the controller tends to select other\nsub-policies to improve fairness. The controller learns the sensible strategy based on the fair-ef\ufb01cient\nreward.\n\n4.2 The Matthew Effect\n\nIn this scenario of the Matthew effect, we investigate whether agents can learn to mitigate/avoid the\nMatthew effect. In the scenario, there are 10 pac-men (agents) initialized with different positions,\nsizes, and speeds and also 3 stationary ghosts initialized at random locations, illustrated in Figure 2\n(mid). Each pac-man can observe the nearest three other pac-men and the nearest one ghost. It could\nmove to one of four directions or stay at current position. Once the distance between the pac-men\nand a ghost is less than the agent\u2019s size, the ghost is consumed and the agent gets a reward 1. Then, a\nnew ghost will be generated at a random location. When the agent gets a reward, its size and speed\nwill increase correspondingly until the upper bounds are reached. In this scenario, the pac-man who\nconsumes more ghosts becomes larger and faster, making consume ghosts easier. So, there exists\ninherent inequality in the setting. We trained all the models for \ufb01ve runs with different random seeds.\nAs pac-men are homogeneous, we let pac-men share weights for all the methods.\n\n7\n\n\fFigure 5: The Matthew effect\n\nFigure 5 shows the performance of FEN and the baselines in terms of social welfare (total ghosts\nconsumed by all the pac-men), CV, and min and max income (consumed ghosts) among pac-men,\nepisodes to converge. The detailed results are available in Appendix. Random pac-men take random\nactions and their CV shows the inherent unfairness of this scenario. Min pac-men cannot learn\nreasonable policies, because min\u03c8 is always closed to 0. Min+\u03b1Avg is only a little fairer than Avg\nsince the effect of min\u03c8 is very weak. Independent causes the Matthew effect as indicated by the\nmin (close to 0) and max (more than 200) income, where pac-man with initial larger size becomes\nlarger and larger and ghosts are mostly consumed by these larger pac-men. Inequity Aversion is\nslightly fairer than Independent but lower social welfare.\nAlthough Independent has the largest pac-man which consumes ghosts faster than others, this does\nnot necessarily mean they together consume ghosts fast. FEN is not only fairer than the baselines\nbut also has the highest social welfare, even higher than Independent. FEN pac-men have similar\nsizes and consume more ghosts than the baselines. This demonstrates FEN is capable of tackling the\nMatthew effect and helps social welfare increase. FEN w/o Hierarchy focuses more on fairness,\nneglecting the ef\ufb01ciency as in the scenario of job scheduling. Moreover, learning without hierarchy is\nmuch slower than FEN in this scenario, as illustrated in Figure 6. FEN w/o Hierarchy takes about\n6000 episodes, while FEN takes only about 300 episodes, con\ufb01rming that the hierarchy indeed speeds\nup the training.\nDoes distributed average consensus affect the perfor-\nmance of FEN? Instead of using the centrally computed\naverage utility, we employ the gossip algorithm to es-\ntimate the average utility, where each agent only ex-\nchanges information with the agents in its observation.\nAs shown in Figure 5, FEN w/ Gossip performs equiv-\nalently to FEN with only slight variation on each perfor-\nmance metric. The learning curve of FEN w/ Gossip\nis also similar to FEN, as illustrated in Figure 6. These\ncon\ufb01rm that FEN can be trained in a fully decentralized\nway.\nDo sub-policies really learn something useful? To an-\nswer this question, after the training of FEN, we keep the\nlearned weights \u03b8 and \u03c61 and replace other sub-policies\nwith a random sub-policy. Once the controller chooses\nother sub-policies instead of \u03c61, the agent will perform\nrandom actions. In this FEN w/ random sub-policy, the min income become lower than FEN and\nCV becomes higher, because the random sub-policy cannot provide fairness behavior the controller\nrequests. To investigate the difference of learned sub-policies and random sub-policy, we \ufb01x the three\nghosts as a triangle at the center of the \ufb01eld and visualize the distribution of an agent\u2019s positions\nunder each sub-policy, as illustrated in Figure 7. It is clear that the learned sub-policies keep away\nfrom the three ghosts for fairness and their distributions are distinct, concentrated at different corners,\nverifying the effect of the information-theoretic reward.\n\nFigure 6: Learning curves of FEN, FEN w/\nGossip, and FEN w/o Hierarchy in the\nMatthew effect.\n\n4.3 Manufacturing Plant\n\nIn this scenario of manufacturing plant, we investigate whether agents with different needs can learn\nto share different types of resources and increase the production in a manufacturing plant. In a 8 \u00d7 8\n\n8\n\n\fFigure 7: Visualization of learned sub-policies (left three) and random sub-policy (right) in the\nMatthew effect.\n\ngrid world, there are 5 agents and 8 gems, as illustrated in Figure 2 (right). The gems belong to\nthree types (y, g, b). Each agent has a local observation that contains a square view with 5 \u00d7 5 grids\ncentered at the agent itself, and could move to one of four neighboring grids or stay. When an agent\nmoves to the grid of one gem, the agent collects the gem and gets a reward of 0.01, and then a new\ngem with random type and random location is generated. The total number of gems is limited, and\nwhen all the gems are collected by the agents, the game ends. Each agent has a unique requirement of\nnumbers for the three types of gems to manufacture a unique part of the product and receive a reward\n1. Each product is assembled by the \ufb01ve unique parts manufactured by the \ufb01ve agents, respectively.\nSo, the number of manufactured products is determined by the least part production among the agents.\nDue to the heterogeneity, we let each agent learn its own weights for FEN and the baselines.\nTable 2 shows the performance of FEN and the baselines in terms of resource utilization (the ratio\nof the number of gems consumed to manufacture the products over the total number of gems), CV,\nnumber of products (minimum number of manufactured parts among agents), and max number of\nmanufactured parts among agents. In this scenario, agents need to learn to collect the right gems and\nthen to balance the parts manufactured by each agent (i.e., manufacturing similar large number of\nparts), because the unused collected gems and redundant parts will be wasted. FEN manufactures\nthe most products, more than two times than the baselines. The more products are assembled, the\nhigher the resource utilization is. Thus, FEN also has the highest resource utilization. Moreover, FEN\nis also the fairest one. Although FEN w/o Hierarchy agents are fairer than other baselines, they all\nmanufacture less parts and hence eventually less products. Avg agents assemble the least products,\nthough one agent manufactures the largest number of parts, resulting in serious waste.\n\nTable 2: Manufacturing plant\n\nresource utilization\n\nCV\n\nno. products max parts\n\n28% \u00b15%\n27% \u00b16%\n29% \u00b16%\n13% \u00b13%\n34% \u00b16%\n82% \u00b15%\n22% \u00b13%\n\n0.38 \u00b10.08\n0.27 \u00b10.06\n0.26 \u00b10.01\n0.63 \u00b10.07\n0.28 \u00b10.01\n0.10 \u00b10.03\n0.18 \u00b10.07\n\n19 \u00b13\n19 \u00b14\n20 \u00b14\n9 \u00b12\n23 \u00b14\n48 \u00b13\n15 \u00b11\n\n58 \u00b18\n42 \u00b17\n41 \u00b17\n71 \u00b19\n45 \u00b17\n63 \u00b13\n24 \u00b14\n\nIndependent\n\nInequity Aversion\n\nFEN w/o Hierarchy\n\nMin\nAvg\n\nFEN\n\nMin+\u03b1Avg\n\n5 Conclusion\n\nWe have proposed FEN, a novel hierarchical reinforcement learning model to learn both fairness and\nef\ufb01ciency, driven by fair-ef\ufb01cient reward, in multi-agent systems. FEN consists of one controller and\nseveral sub-policies, where the controller learns to optimize the fair-ef\ufb01cient reward, one sub-policy\nlearns to optimize the environmental reward, and other sub-policies learn to provide diverse fairness\nbehaviors guided by the derived information-theoretic reward. FEN can learn and execute in a fully\ndecentralized way, coordinated by average consensus. It is empirically demonstrated that FEN easily\nlearns both fairness and ef\ufb01ciency and signi\ufb01cantly outperforms baselines in a variety of multi-agent\nscenarios including job scheduling, the Matthew effect, and manufacturing plant.\n\nAcknowledgments\n\nThis work was supported in part by Huawei Noah\u2019s Ark Lab, Peng Cheng Lab, and NSF China under\ngrant 61872009.\n\n9\n\n\fReferences\n[1] Jiechuan Jiang, Chen Dun, and Zongqing Lu. Graph convolutional reinforcement learning for multi-agent\n\ncooperation. arXiv preprint arXiv:1810.09202, 2018.\n\n[2] Bram Bakker, Shimon Whiteson, Leon Kester, and Frans CA Groen. Traf\ufb01c light control by multiagent\nreinforcement learning systems. In Interactive Collaborative Information Systems, pages 475\u2013510. 2010.\n\n[3] Dorian Minarolli and Bernd Freisleben. Virtual machine resource allocation in cloud computing via\n\nmulti-agent fuzzy control. In International Conference on Cloud and Green Computing, 2013.\n\n[4] Steven de Jong, K Tuyls, Katja Verbeeck, and Nico Roos. Considerations for fairness in multi-agent\n\nsystems. In ALAMAS, pages 104\u2013110. 2007.\n\n[5] Ariel D Procaccia. Thou shalt covet thy neighbor\u2019s cake. In International Joint Conference on Arti\ufb01cial\n\nIntelligence (IJCAI), 2009.\n\n[6] Yiling Chen, John K Lai, David C Parkes, and Ariel D Procaccia. Truth, justice, and cake cutting. Games\n\nand Economic Behavior, 77(1):284\u2013297, 2013.\n\n[7] Ian Kash, Ariel D Procaccia, and Nisarg Shah. No agent left behind: Dynamic fair division of multiple\n\nresources. Journal of Arti\ufb01cial Intelligence Research, 51:579\u2013603, 2014.\n\n[8] Chongjie Zhang and Julie A Shah. Fairness in multi-agent sequential decision-making. In Advances in\n\nNeural Information Processing Systems (NeurIPS), 2014.\n\n[9] Aur\u00e9lie Beynier, Nicolas Maudet, and Anastasia Damamme. Fairness in multiagent resource allocation\nwith dynamic and partial observations. In International Joint Conference on Autonomous Agents and\nMultiagent Systems (AAMAS), 2018.\n\n[10] Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, OpenAI Pieter Abbeel, and Igor Mordatch. Multi-agent actor-\ncritic for mixed cooperative-competitive environments. In Advances in Neural Information Processing\nSystems (NeurIPS), 2017.\n\n[11] Sainbayar Sukhbaatar, Rob Fergus, et al. Learning multiagent communication with backpropagation. In\n\nAdvances in Neural Information Processing Systems (NeurIPS), 2016.\n\n[12] Yaodong Yang, Rui Luo, Minne Li, Ming Zhou, Weinan Zhang, and Jun Wang. Mean \ufb01eld multi-agent\n\nreinforcement learning. In International Conference on Machine Learning (ICML), 2018.\n\n[13] Jiechuan Jiang and Zongqing Lu. Learning attentional communication for multi-agent cooperation.\n\nAdvances in Neural Information Processing Systems (NeurIPS), 2018.\n\n[14] Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojciech Marian Czarnecki, Vinicius Zambaldi, Max\nJaderberg, Marc Lanctot, Nicolas Sonnerat, Joel Z Leibo, Karl Tuyls, et al. Value-decomposition networks\nfor cooperative multi-agent learning. arXiv preprint arXiv:1706.05296, 2017.\n\n[15] Tabish Rashid, Mikayel Samvelyan, Christian Schroeder de Witt, Gregory Farquhar, Jakob Foerster, and\nShimon Whiteson. Qmix: Monotonic value function factorisation for deep multi-agent reinforcement\nlearning. arXiv preprint arXiv:1803.11485, 2018.\n\n[16] Jakob Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Counter-\n\nfactual multi-agent policy gradients. In AAAI Conference on Arti\ufb01cial Intelligence (AAAI), 2018.\n\n[17] Rahul Jain, Preeti Ranjan Panda, and Sreenivas Subramoney. Cooperative multi-agent reinforcement\nlearning-based co-optimization of cores, caches, and on-chip network. ACM Transactions on Architecture\nand Code Optimization, 14(4):32, 2017.\n\n[18] Jingjing Cui, Yuanwei Liu, and Arumugam Nallanathan. Multi-agent reinforcement learning based resource\n\nallocation for uav networks. arXiv preprint arXiv:1810.10408, 2018.\n\n[19] Xihan Li, Jia Zhang, Jiang Bian, Yunhai Tong, and Tie-Yan Liu. A cooperative multi-agent reinforcement\nlearning framework for resource balancing in complex logistics network. arXiv preprint arXiv:1903.00714,\n2019.\n\n[20] Alexander Peysakhovich and Adam Lerer. Prosocial learning agents solve generalized stag hunts better\nthan sel\ufb01sh ones. In International Conference on Autonomous Agents and MultiAgent Systems (AAMAS),\n2018.\n\n10\n\n\f[21] Edward Hughes, Joel Z Leibo, Matthew Phillips, Karl Tuyls, Edgar Due\u00f1ez-Guzman, Antonio Garc\u00eda\nCasta\u00f1eda, Iain Dunning, Tina Zhu, Kevin McKee, Raphael Koster, et al. Inequity aversion improves\ncooperation in intertemporal social dilemmas. In Advances in Neural Information Processing Systems\n(NeurIPS), 2018.\n\n[22] Jane X Wang, Edward Hughes, Chrisantha Fernando, Wojciech M Czarnecki, Edgar A Du\u00e9\u00f1ez-Guzm\u00e1n,\nand Joel Z Leibo. Evolving intrinsic motivations for altruistic behavior. In International Conference on\nAutonomous Agents and MultiAgent Systems (AAMAS), 2019.\n\n[23] Thijs Bol, Mathijs de Vaan, and Arnout van de Rijt. The matthew effect in science funding. Proceedings of\n\nthe National Academy of Sciences, 115(19):4887\u20134890, 2018.\n\n[24] Matja\u017e Perc.\n\nThe matthew effect in empirical data.\n\n11(98):20140378, 2014.\n\nJournal of The Royal Society Interface,\n\n[25] Yichuan Jiang and Zhichuan Huang. The rich get richer: Preferential attachment in the task allocation of\ncooperative networked multiagent systems with resource caching. IEEE Transactions on Systems, Man,\nand Cybernetics-Part A: Systems and Humans, 42(5):1040\u20131052, 2012.\n\n[26] Yulong Zeng and Song Zuo. The matthew effect in computation contests: High dif\ufb01culty may lead to 51%\n\ndominance. arXiv preprint arXiv:1902.09089, 2019.\n\n[27] Zhang-Wei Hong, Shih-Yang Su, Tzu-Yun Shann, Yi-Hsiang Chang, and Chun-Yi Lee. A deep policy\ninference q-network for multi-agent systems. In International Conference on Autonomous Agents and\nMultiAgent Systems (AAMAS), 2018.\n\n[28] Neil C Rabinowitz, Frank Perbet, H Francis Song, Chiyuan Zhang, SM Eslami, and Matthew Botvinick.\n\nMachine theory of mind. arXiv preprint arXiv:1802.07740, 2018.\n\n[29] Peter Dayan and Geoffrey E Hinton. Feudal reinforcement learning. In Advances in Neural Information\n\nProcessing Systems (NeurIPS), 1993.\n\n[30] Alexander Sasha Vezhnevets, Simon Osindero, Tom Schaul, Nicolas Heess, Max Jaderberg, David Silver,\nIn International\n\nand Koray Kavukcuoglu. Feudal networks for hierarchical reinforcement learning.\nConference on Machine Learning (ICML), 2017.\n\n[31] O\ufb01r Nachum, Shixiang Shane Gu, Honglak Lee, and Sergey Levine. Data-ef\ufb01cient hierarchical reinforce-\n\nment learning. In Advances in Neural Information Processing Systems (NeurIPS), 2018.\n\n[32] Richard S Sutton, Doina Precup, and Satinder Singh. Between mdps and semi-mdps: A framework for\n\ntemporal abstraction in reinforcement learning. Arti\ufb01cial intelligence, 112(1-2):181\u2013211, 1999.\n\n[33] Pierre-Luc Bacon, Jean Harb, and Doina Precup. The option-critic architecture. In AAAI Conference on\n\nArti\ufb01cial Intelligence (AAAI), 2017.\n\n[34] Kevin Frans, Jonathan Ho, Xi Chen, Pieter Abbeel, and John Schulman. Meta learning shared hierarchies.\n\narXiv preprint arXiv:1710.09767, 2017.\n\n[35] Rajendra K Jain, Dah-Ming W Chiu, and William R Hawe. A quantitative measure of fairness and\n\ndiscrimination. Technical report, 1984.\n\n[36] Lin Xiao, Stephen Boyd, and Seung-Jean Kim. Distributed average consensus with least-mean-square\n\ndeviation. Journal of parallel and distributed computing, 67(1):33\u201346, 2007.\n\n[37] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy\n\noptimization algorithms. arXiv preprint arXiv:1707.06347, 2017.\n\n11\n\n\fAppendix\n\nHyperparameters\n\nIn the experiments, we use PPO for every RL agent. The PPO structure keeps same for the controller and\nsub-policies of FEN, and also for the baselines. The value network and policy network are MLPs with two\n256-unit hidden layers and ReLU activation. The learning rates for the value network and policy network of\nPPO are 10\u22123 and 3 \u00d7 10\u22124, respectively. We trained all networks using Adam optimizer. The discounted\nfactor is \u03b3 = 0.98. For Min+\u03b1Avg agents, \u03b1 = 0.01. For Inequity Aversion agents, \u03b1 = 5 and \u03b2 = 0.05,\nthe setting used in [21]. For FEN, the number of the sub-policies is 4, \u0001 in the fair-ef\ufb01cient reward is set to 0.1.\nT is 25, 50, and 50 in job scheduling, the Matthew effect, and manufacturing plant, respectively.\n\nExperimental Settings\n\nIn the scenario of job scheduling, we trained all the models for \ufb01ve runs with different random seeds, each\nepisode contains 1000 timesteps.\nIn the scenario of the Matthew effect, the size and speed of pac-man are randomly initiated between (0.01, 0.04)\nand (0.018, 0.042), respectively. Each time after a pac-men consumes a ghost, its size will increase by 0.005\nand the speed will increase by 0.004. The max size is 0.15 and the max speed is 0.13. We trained all the models\nfor \ufb01ve runs with different random seeds, where each episode contains 1000 timesteps.\nIn the scenario of manufacturing plant, the total number of gems is 700. The unique requirements of numbers\nof three types of gems (Ly, Lg, Lb) are (2, 1, 0), (1, 0, 1), (0, 1, 1), (1, 1, 0), (0, 1, 2), respectively, for the \ufb01ve\nagents. We trained all the models for \ufb01ve runs with different random seeds, where each episode ends after all the\ngems are collected by agents.\nAll the experiments were conducted on Dell XPS desktops with i7-8700k 6-Core processors and 1080TI GPUs.\n\nExperimental Results\n\nThe details of the experimental results in the Matthew effect are shown in Table 3.\n\nTable 3: The Matthew effect\n\nsocial welfare\n\nCV\n\nmin income max income episodes\n\nRandom\n\nIndependent\n\nInequity Aversion\n\nMin\nAvg\n\nFEN\n\nMin+\u03b1Avg\n\nFEN w/ Gossip\n\nFEN w/o Hierarchy\n\nFEN w/ Random Sub-policy\n\n22 \u00b110\n202 \u00b133\n152 \u00b118\n\n7\u00b14\n\n126\u00b118\n103\u00b116\n94\u00b13\n95\u00b14\n26\u00b11\n99\u00b16\n\n-\n\n100\n1000\n6000\n1000\n2000\n300\n300\n6000\n\n-\n\n84 \u00b130\n791 \u00b162\n702 \u00b190\n18 \u00b18\n527\u00b1113\n441\u00b175\n830\u00b122\n841\u00b155\n251\u00b112\n834\u00b147\n\n0.93 \u00b10.25\n0.86 \u00b10.11\n0.80 \u00b10.16\n2.04 \u00b10.66\n0.86 \u00b10.21\n0.85 \u00b10.18\n0.06 \u00b10.01\n0.07 \u00b10.01\n0.06 \u00b10.04\n0.08 \u00b10.02\n\n1\n1\n2\n0\n2\n1\u00b11\n79\u00b12\n76\u00b14\n23\u00b12\n66\u00b17\n\n12\n\n\f", "award": [], "sourceid": 7755, "authors": [{"given_name": "Jiechuan", "family_name": "Jiang", "institution": "Peking University"}, {"given_name": "Zongqing", "family_name": "Lu", "institution": "Peking University"}]}