{"title": "Epsilon-Best-Arm Identification in Pay-Per-Reward Multi-Armed Bandits", "book": "Advances in Neural Information Processing Systems", "page_first": 2880, "page_last": 2890, "abstract": "We study epsilon-best-arm identification, in a setting where during the exploration phase, the cost of each arm pull is proportional to the expected future reward of that arm. We term this setting Pay-Per-Reward.  We provide an algorithm for this setting, that with a high probability returns an epsilon-best arm, while incurring a cost that depends only linearly on the total expected reward of all arms, and does not depend at all on the number of arms. Under mild assumptions, the algorithm can be applied also to problems with infinitely many arms.", "full_text": "\u0001-Best-Arm Identi\ufb01cation in Pay-Per-Reward\n\nMulti-Armed Bandits\n\nSivan Sabato\n\nDepartment of Computer Science\nBen-Gurion University of the Negev\n\nBeer-Sheva, Israel 8410501\nsabatos@cs.bgu.ac.il\n\nAbstract\n\nWe study \u0001-best-arm identi\ufb01cation, in a setting where during the exploration phase,\nthe cost of each arm pull is proportional to the expected future reward of that arm.\nWe term this setting Pay-Per-Reward. We provide an algorithm for this setting, that\nwith a high probability returns an \u0001-best arm, while incurring a cost that depends\nonly linearly on the total expected reward of all arms, and does not depend at all\non the number of arms. Under mild assumptions, the algorithm can be applied also\nto problems with in\ufb01nitely many arms.\n\n1\n\nIntroduction\n\nConsider placing an ad next to search engine results, based on the search query. In a preliminary\nsurvey for a future promotion, a retailer wishes to identify the best query expression to link to its ad,\nthat is, the expression that maximizes the expected number of clicks on the ad. The payment per ad is\nproportional to the number of clicks. However, during the survey, clicks do not lead to pro\ufb01t, while\nthe payment is still due.\nThis problem can be formulated as a type of best-arm-identi\ufb01cation problem in a stochastic multi-\narmed bandit (MAB) setting [see, e.g., 14, 15, 8, 3]. In stochastic MAB, there is a set of arms, each\nassociated with a non-negative reward distribution. Each pull of an arm draws an instantaneous\nreward from the corresponding reward distribution. The MAB algorithm iteratively pulls arms and\nobserves their instantaneous rewards. At the end of the run, the algorithm selects one arm. The\ngoal is to \ufb01nd an arm with a near-maximal expected reward with a high probability. In the example\nabove, each arm in the MAB framework can be mapped to a speci\ufb01c search expression, and pulling\nthe arm is equivalent to linking the ad to the expression represented by this arm for a given period\nof time, and then observing the number of clicks. In standard MAB, each arm pull incurs a unit\ncost. However, in the example above, the cost of pulling an arm is the number of clicks on the ad,\nassuming the common pay-per-click advertisement model. Thus, the expected cost in the survey stage\nis proportional to the expected reward after the survey stage. We term this setting Pay-Per-Reward\n(PPR). Other applications of this model include any case where there is an exploration stage in which\nrewards are not collected, but payment is still proportional to the quality of the arm. For instance,\nconsider a sensor \ufb01eld, where the goal is to \ufb01nd the most active sensor, but each activity report has\na communication cost. Here, \u201cpulling an arm\u201d is done by activating the sensor\u2019s reporting option\nfor a given period of time. Another example is \ufb01nding the best crowd worker on a crowd-sourcing\nplatform. Here, an \"arm pull\" is equivalent to providing the worker with a test task. The workers are\npaid according to their success rate in the test task, thus a more successful worker will be paid more\nduring the exploration stage.\nOur contribution. We provide an algorithm, MAB-PPR, for the pay-per-reward setting, that \ufb01nds\nan \u0001-optimal arm with a probability of at least 1 \u2212 \u03b4, for given \u0001, \u03b4 \u2208 (0, 1). We show that the cost\n\n33rd Conference on Neural Information Processing Systems (NeurIPS 2019), Vancouver, Canada.\n\n\fincurred by MAB-PPR has no dependence on the number of arms, and only a linear dependence on\nthe total expected reward of all arms. Our results generalize beyond bounded reward distributions,\nand support some heavy-tailed distributions with bounded second moments. Under mild assumptions,\nthe algorithm can be applied also to problems with in\ufb01nitely many arms.\nSince the cost of MAB-PPR is independent of the number of arms, it is especially useful when the\nnumber of arms is potentially very large, while the total expected reward is bounded. For instance, in\nthe ad-placement example, the number of arms is proportional to the number of admissible search\nexpressions, which can be practically unbounded, but only a small number of those are expected to\nhave large click rates when linked to a speci\ufb01c ad. Other applications in which the number of arms\ncan be very large include, for instance, recommender systems [32], content personalization [24] and\nweb-content optimization [1].\nFor the standard setting, in which each arm pull has a unit cost, [14] provided an \u0001-best-arm\nidenti\ufb01cation algorithm with a sample complexity that depends linearly on the number of arms,\nusing an algorithm that they term the median-elimination algorithm. This algorithm halves the\nnumber of candidate arms in each round based on their empirical mean rewards. Instead, in our\nsetting, one should try to halve the total expected reward of the arms that remain in each round. This\nrequires a new estimation scheme. Another challenge in the PPR setting is that due to the potentially\nunbounded number of arms, sums of instantaneous rewards of sets of arms do not have a bounded\nsupport, even if single-arm rewards are bounded. Moreover, in some settings one might prefer to\nsimultaneously pull an entire set of arms from the joint distribution of rewards of all arms. Thus,\nstandard assumptions on independence between arms do not necessarily hold.\nMAB-PPR addresses these challenges using a two-step estimation procedure in each round, combined\nwith heavy-tailed mean estimation. This guarantees success even under some dependence between\narms. Support for some unbounded reward distributions of single arms is easily obtained within this\nframework, by using heavy-tailed mean estimation for the expected rewards of single arms as well.\nOur analysis shows that the PPR setting can be successfully handled in a wide range of regimes.\n\nPaper structure After discussing related work, we present the setting and preliminaries in Section 2.\nThe main result is stated in Section 3. The MAB-PPR algorithm is listed and described in Section 4.\nWe prove the main result in Section 5, and conclude in Section 6.\n\n1.1 Related work\n\n[14] formulated the stochastic MAB \u0001-best-arm identi\ufb01cation problem (also termed the PAC setting)\nfor bounded rewards. They showed that the number of pulls can be linear in the number of arms,\nwithout logarithmic factors that would be necessary in a naive algorithm that pulls all arms the same\nnumber of times. Closely matching lower bounds for this setting were given in [22]. Instance-speci\ufb01c\nbounds in this setting were studied in [19, 17, 12]. A \ufb01xed-budget variant was studied in [8, 3, 10].\nThe works above assume the standard cost model for MAB, in which each arm pull costs one unit. A\ndifferent cost model has been studied in the setting of budgeted MAB [25, 26, 13, 27, 29, 20]. Here,\narms are associated with a random or deterministic cost, in addition to a reward distribution, and\nthe goal is to maximize the cumulative reward within total cost constraints. [30] studied a best-arm\nvariant of this problem. MAB with in\ufb01nitely many arms, also termed many-armed bandits, has been\nstudied in the unit-cost model both for the cumulative regret setting [5, 6, 28, 20] and for exploration\nsettings [11, 4]. [20] study in\ufb01nitely-many arms in the budgeted MAB setting.\nA useful tool for mean estimation for heavy-tailed random variables is median-of-means estimation\n[2]. This tool has been recently used in various contexts in machine learning [e.g., 23, 7, 16, 21, 18].\nMAB with heavy-tailed rewards under the standard unit-cost model was studied in the regret setting\n[9] and for pure exploration with unbounded rewards [31].\n\n2 Setting and Preliminaries\nFor an integer i, denote [i] := {1, . . . , i}. Let K be the total number of available arms, and denote the\nset of available arms by [K].1 Let X1, . . . , XK be the random variables associated with the reward\n\n1We discuss in Section 4 how to use MAB-PPR when the set of available arms is in\ufb01nite or unknown.\n\n2\n\n\fdistributions of the corresponding arms. We assume that the random variables are non-negative, and\nhave a \ufb01nite expectation and variance. Pulling arm i is equivalent to drawing an independent copy of\nXi and observing its value. We study here a more general setting, in which sets of arms can be pulled\nsimultaneously. We assume some joint distribution of the random vector X = (X1, . . . , XK). A pull\nof a speci\ufb01c set of arms is equivalent to drawing an independent copy of X and observing the rewards\nof all the arms in the set. The standard setting, in which arms are pulled one by one, is equivalent to\nassuming that X1, . . . , XK are statistically independent in X. Thus, we generally assume below that\nsets are simultaneously pulled.\nDenote the expected reward of arm i by wi := E[Xi], termed below the arm\u2019s weight. Let W :=\ni\u2208A wi. Consider a run of\nthe algorithm in which the total number of times each arm is pulled (regardless of whether some of\nthe arms were pulled simultaneously) is n, and let I1, . . . , In be the sequence of pulled arms. The\ni\u2208[n] wIi. We make two assumptions on the distribution of X.\ni := Var(Xi) \u2264 wi \u2264 1.\n\n(cid:80)\ni\u2208[K] wi. For a set A \u2286 [K], denote XA :=(cid:80)\ncost of the algorithm in this run is(cid:80)\n\ni\u2208A Xi and WA :=(cid:80)\n\nAssumption 2.1. For all i \u2208 [K], \u03c32\n\ni\u2208A \u03c32\n\ni \u2264 (cid:80)\n\nA = (cid:80)\n\nThis assumption clearly holds if Xi are bounded in [0, 1]. Nonetheless, it is more general, and can\nhold also for reward distributions with unbounded support.\nAssumption 2.2. There is a constant V \u2265 0, such that for any subset of the arms A \u2286 [K],\nA := Var(XA) \u2264 V WA.\n\u03c32\nAssumption 2.2 trivially holds with V = 1 if arm-pulls of individual arms are statistically independent,\nas when they are pulled separately. In this case, \u03c32\ni\u2208A wi = WA, where\nAssumption 2.1 was used to bound \u03c32\ni . If arm rewards are not statistically independent, then the\nexistence and value of the constant V depend on the distribution of X. For instance, if each arm is\npositively correlated with at most Q other arms, then V can be upper bounded by 1 + 2Q (see the\nsupplementary material, Appendix A). We note that our results below are indifferent to values of V\nas large as 1\n\u0001 .\nWe use the median-of-means method [2] to estimate the mean of a distribution with a bounded\nvariance. In this method, q batches of (cid:96) independent samples are used; A separate empirical mean is\ncalculated from each batch, and the estimate is the median of these means. Formally, the (q, (cid:96))-MoM\nestimator of a random variable X from a sample of i.i.d. draws {xi,j}i\u2208[q],j\u2208[(cid:96)] of X, is de\ufb01ned\nas Median{ 1\nj\u2208[(cid:96)] xi,j | i \u2208 [q]}. We use the following formulation of the result of [2] on the\nconvergence of the MoM estimator to the true mean, similarly to [16]. The proof is provided for\ncompleteness in the supplementary material, Appendix B.\nLemma 2.3 (Median-of-Means). Let X be a random variable with mean w and variance \u03c32 < \u221e.\nLet q, (cid:96) be integers, and let \u02c6w be the (q, (cid:96))-MoM estimate of w from a sample of q(cid:96) i.i.d. draws of\ncopies of X. With a probability at least 1 \u2212 exp(\u2212 2\n\n9 q), we have |w \u2212 \u02c6w| \u2264(cid:112)6\u03c32/(cid:96).\n\n(cid:80)\n\n(cid:96)\n\n3 Main result\nWe propose the MAB-PPR algorithm, which accepts parameters \u0001, \u03b4 \u2208 (0, 1), iteratively pulls arms,\nand outputs a single arm. MAB-PPR satis\ufb01es the following guarantee.\n\nTheorem 3.1. Suppose that Assumptions 2.1 and 2.2 hold. Let W := (cid:80)\n\ni\u2208[K] wi, and let V as\nde\ufb01ned in Assumption 2.2. With a probability of at least 1 \u2212 \u03b4, MAB-PPR returns an \u0001-optimal arm,\nand its cost is upper bounded by\n\n(cid:18)\n\nW log(1/\u03b4) \u00b7 max\n\nO\n\n(cid:18) 1\n\nV\n\u0001\n\n\u00012 ,\n\n(cid:19)(cid:19)\n\n.\n\nHere and below, O(\u00b7) stands for universal multiplicative constants. In comparison, previous algorithms\nfor the unit-cost model, such as the ones studied in [14], would cost here at least \u2126(W log(K/\u03b4)/\u00012)\nwhere K is the number of arms, assuming bounded rewards. Qualitatively, removing the dependence\non K is of signi\ufb01cance. MAB-PPR obtains a quantitatively signi\ufb01cant improvement when K is large\nand V \u2264 O(1/\u0001).\n\n3\n\n\f(0, 1).\n\nq, C1, C2, C3, N\ufb01nal > 0; \u03c1, \u03b8 \u2208\n\nq/\u03b4t) and (cid:96)t \u2190 C1 max(1/\u00012\n\nAlgorithm 1 MAB-PPR: \u0001-Best-Arm-Identi\ufb01cation with Pay-Per-Reward\nInput: \u0001, \u03b4 \u2208 (0, 1), V \u2265 1, K \u2208 N; Universal constants Cq, C(cid:48)\nOutput: r \u2208 [K]\n1: t \u2190 1, S1 \u2190 [K], \u00011 \u2190 (1 \u2212 \u03c1)\u0001/2, \u03b41 \u2190 \u03b4/2.\n2: while |St| > N\ufb01nal do\n(cid:66) First batch of pulls:\n3:\nSet q1 \u2190 Cq log(C(cid:48)\nt , V /\u0001t); Pull the arms in St for q1(cid:96)t times.\n4:\nFor each i \u2208 St, use the pulls of arm i to calculate \u02c6wi,t, the (q1, (cid:96)t)-MoM estimate of wi.\n5:\nSet {jl}l\u2208[|St|] to be an ordering of St such that \u02c6wjl,t \u2264 \u02c6wjl+1,t for all l \u2208 [|St| \u2212 1].\n6:\n(cid:66) Second batch of pulls:\n7:\nSet q2 \u2190 9\n8:\nFor i \u2208 St, let {xj,l\n9:\nfor l \u2208 [|St|] do\n10:\n11:\n12:\n13:\n14:\n15:\n16:\n17:\n18: end while\n19: (cid:66) Final round (|St| \u2264 N\ufb01nal):\n20: Set q3 \u2190 9\n21: For i \u2208 St, use the pulls of arm i to calculate \u02c6wi,t, the (q3, (cid:96)(3))-MoM estimate of wi.\n22: return some r \u2208 argmaxi\u2208St \u02c6wi,t.\n\nUse {(cid:80)\nIf(cid:102)WSt,t \u2264 \u0001/4, return some r \u2208 St and terminate.\nFind the largest integer Nt such that(cid:102)WANt ,t/(cid:102)WSt,t \u2264 \u03b8.\nSet St+1 \u2190 St \\ ANt, \u0001t+1 \u2190 \u03c1\u0001t, \u03b4t+1 \u2190 \u03b4t/2, t \u2190 t + 1.\n\n2 log(N\ufb01nal/\u03b4t), (cid:96)(3) \u2190 C3/\u00012; Pull the arms in St for q3(cid:96)(3) times.\n\nend for\n(cid:66) Decide which arms to remove:\n\n2 log(12/\u03b4t), (cid:96)(2) \u2190 C2V /\u0001; Pull the arms in St for q2(cid:96)(2) times.\ni }j\u2208[q2],l\u2208[(cid:96)(2)] be the i.i.d. copies of Xi obtained in line 8.\ni }j\u2208[q2],l\u2208[(cid:96)(2)] to calculate(cid:102)WAl,t, the (q2, (cid:96)(2))-MoM estimate of WAl.\n\nxj,l\n\nDe\ufb01ne Al := {j1, . . . , jl}.\n\ni\u2208Al\n\n4 The MAB-PPR algorithm\n\nThe MAB-PPR algorithm is listed in Alg. 1. The main ideas in MAB-PPR are described below. We\nuse several universal constants in Alg. 1. In the analysis below, we show that there exist values for\nthese constants such that the guarantees of MAB-PPR hold. For readability, we do not explicitly\nround up non-integer sample sizes. Such rounding would not affect our \ufb01nal cost upper bound.\nA central challenge in designing MAB-PPR is avoiding a union bound on the number of arms. In\n[14], where arm pulls have unit cost, the median-elimination algorithm was proposed for this purpose.\nThis algorithm removes half of the arms from the candidate set in each round. In our case, because of\nthe different cost model, it is necessary to remove a constant fraction of the total weight, regardless\nof the number of arms. To achieve this, MAB-PPR uses two batches of arm pulls in each round.\nThe universal constants Cq, C(cid:48)\nq, C1, C2 are used for calculating the required numbers of arm pulls\nin each batch so that subsequent estimates hold. The \ufb01rst batch of pulls (line 4) is used to estimate\nthe weight of each arm with the MoM estimator (line 5). The arms are then ordered according to\ntheir estimated weight. The second batch of pulls (line 8) is used to decide how many arms to remove\nfrom the bottom of this ordering, to make sure that the removed fraction of the total weight is within a\ncertain range. This is done by estimating the total weight of each possible subset of arms, again using\nMoM (line 12). Here, it is important to note that taking the sum of MoM estimates is not equivalent\nto taking the MoM estimate of the sum, thus these sums are estimated directly. The selected number\nof arms is then removed from the bottom of the ordering (line 16). The threshold \u03b8 is used to upper\nbound the empirical fraction of the removed arms. After each round, the values of \u0001t and \u03b4t are\nupdated, where the constant \u03c1 controls the rate of decay for \u0001t. MAB-PPR terminates in one of\ntwo ways: if at some point, the estimated total weight of the remaining arms is small, then some\nremaining arm is returned (line 15). Otherwise, when the number of remaining arms is smaller than\nthe constant N\ufb01nal, all their weights are directly estimated (where the constant C3 is used to set the\nnumber of pulls), and the arm with the maximal estimated weight is returned (line 22).\n\n4\n\n\fin an ad-click. Assuming that W \u2261(cid:80)\n\nSupporting an unbounded number of arms. For simplicity of presentation, the notations and\nAlg. 1 are given for a \ufb01nite and known number of arms K. Nonetheless, MAB-PPR can be used\neven if the number of arms is unbounded or in\ufb01nite. This is possible if there is a way to pull all\narms together in \ufb01nite time, and get a \ufb01nite result listing all the non-zero instantaneous rewards. For\ninstance, in the search-query example, pulling all arms together can be done by placing an ad and\nlinking it to all search queries, and then recording which search queries actually came up and resulted\ni wi < \u221e, Alg. 1 can then support an unbounded number of\narms as follows. In line 1, S1 should be (implicitly) set to the entire set of available arms. In line 6\nof the \ufb01rst round, only arms with a non-zero reward in any of the pulls of the \ufb01rst batch need to be\nexplicitly ordered. The other arms can be implicitly placed at the bottom of the ordering. After the\nsecond batch, let l0 be the smallest index such that arm jl0 got a positive reward at least once. Then\nround, satis\ufb01es N1 \u2265 l0. Thus, in the \ufb01rst round, any arm that got no reward in any of the pulls is\nremoved in line 17. As a result, the set of arms S2 set for the second round is \ufb01nite and known, and\nso from this point on, the algorithm can proceed as listed.\n\nfor all l < l0,(cid:102)WAl,1 can be (implicitly) set to zero. Clearly, the value of N1, set in line 16 of the \ufb01rst\n\n5 Analysis\n\nIn this section we prove Theorem 3.1, restated below in more detail as Theorem 5.6. Recall that St is\nthe set of candidate arms in round t. Denote by Wt := WSt the total weight of the arms in St. As\n\nin Alg. 1, the estimations made by MAB-PPR are denoted by { \u02c6wi,t}i\u2208St and {(cid:102)WAl,t}l\u2208[|St|]. We\nomit the subscript t when it is clear from context. Denote(cid:102)Wt :=(cid:102)WSt,t. We say that a round t of\n\nMAB-PPR is standard if |St| > N\ufb01nal and Wt \u2265 \u0001/8. We \ufb01rst show that with a high probability, the\nonly non-standard round (if at all) is the last round. De\ufb01ne the following event:\n\nE0(t) := {(Wt < \u0001/8 \u21d2(cid:102)Wt \u2264 \u0001/4) and (Wt \u2265 \u0001/2 \u21d2(cid:102)Wt \u2265 \u0001/4)}.\n\nLemma 5.1. For a large enough constant C2, for any round t with |St| > N\ufb01nal, E0(t) holds with a\nprobability at least 1 \u2212 \u03b4t/4.\n\nProof. If |St| > N\ufb01nal then line 8 is invoked, and (cid:102)Wt is a (q2, (cid:96)(2))-MoM estimate of Wt. By\nleast 1 \u2212 \u03b4t/4, |Wt \u2212(cid:102)Wt| \u2264(cid:112)6V Wt/(cid:96)(2). We have (cid:96)(2) = C2V /\u0001. By setting C2 \u2265 48, we get\n|Wt \u2212(cid:102)Wt| \u2264(cid:112)Wt\u0001/8. If Wt < \u0001/8, it follows that(cid:102)Wt \u2264 Wt + \u0001/8 \u2264 \u0001/4, which shows that the\n\ufb01rst part of E0(t) holds. If Wt \u2265 \u0001/2, then(cid:102)Wt \u2265 Wt \u2212(cid:112)Wt\u0001/8 \u2265 Wt/2 \u2265 \u0001/4, which shows that\n\n\u2264 V Wt. Hence, by Lemma 2.3, since q2 \u2265 9\n\n2 log(4/\u03b4t), with a probability at\n\nAssumption 2.2, \u03c32\nSt\n\nthe second part of E0(t) holds.\n\nIf E0(t) holds and Wt < \u0001/8, then in round t the condition in line 15 is invoked, leading to the\ntermination of MAB-PPR. The other option for a round t to be non-standard is if |St| \u2264 N\ufb01nal. This\ncase also leads to the termination of the algorithm. Therefore, if E0(t) holds for all rounds, then all\nrounds except for (perhaps) the last one are standard. We use this observation in the proofs below.\nWe prove the correctness of MAB-PPR by guaranteeing that with a high probability, an \u0001-best arm\nremains in the set St throughout the algorithm. We upper bound the cost of MAB-PPR by showing\nthat under the same events, the total weight of arms is reduced in every round by at least a certain\nfraction. This general analysis structure is also used for the median-elimination algorithm in [14],\nwhere half of the arms are removed in each round. However, once weights are used instead of\nnumbers of arms, new techniques are required, and the estimation scheme, as well as the analysis,\nbecome more involved.\nFirst, we de\ufb01ne several events. We later prove that they all hold together with a high probability.\nLet i\u2217\nt ,t.\nt := \u02c6wi\u2217\n1. For any arm j \u2208 St, we say\nNote that i\u2217 := i\u2217\nt \u2212 \u0001t, where \u0001t is as de\ufb01ned in Alg. 1. We also de\ufb01ne the set\nthat j is a t-bad arm if wj < w\u2217\nMt := {i \u2208 St | i is t-bad and \u02c6wi,t \u2265 \u02c6w\u2217\nt }.\nWe de\ufb01ne three constants: \u03c6L, \u03c6M , \u03c6U \u2208 (0, 1) (standing for \u201cLower\u201d, \u201cMiddle\u201d and \u201cUpper\u201d), such\nthat \u03c6L < \u03c6M < 1 \u2212 \u03c6U . These constants are used to analyze the fraction of removed arms in\n\nt \u2208 argmaxi\u2208St wi be an arm with the largest weight in St. Let w\u2217\n\n1 is the optimal arm and its weight is w\u2217 := w\u2217\n\nt , and \u02c6w\u2217\n\nt := wi\u2217\n\n5\n\n\feach round, as follows: \u03c6L is a lower-bound on the fraction of the arm weight which is guaranteed\nto be removed in each round. \u03c6U is an upper bound on the fraction of the arm weight in bad arms.\n\u03c6M is used to de\ufb01ne a set of arms guaranteed to be removed in each round. Denote the set of arms\n(cid:80)\nthat appear to be the worst in round t according to the estimates { \u02c6wi,t} by Lt = {j1, . . . , jkt},\nwhere {jl}l\u2208[|St| is the ordering de\ufb01ned in line 6 of round t, and kt is the largest integer such that\nwi \u2264 \u03c6M Wt. We de\ufb01ne the following events for each round t, where \u03c6L, \u03c6U \u2208 (0, 1) are\nconstants such that :\n\ni\u2208Lt\n\n\u2022 E1(t) := {WMt < \u03c6U Wt},\n\u2022 E2(t) := {WLt \u2265 \u03c6LWt or |St \\ Lt| \u2264 N\ufb01nal},\n\u2022 E3(t) := {(St+1 \\ Mt (cid:54)= \u2205) \u2227 (St+1 \u2229 Lt = \u2205).\n\nWe omit the argument (t) on events when it is clear from context. The following lemmas state that\nin a standard round t, all of these events hold with a high probability, as long as the constants are\nselected appropriately. Their proofs are provided after the statement and proof of Theorem 5.6.\nLemma 5.2. For any \u03c6U \u2208 (0, 1) and suf\ufb01ciently large constants Cq, C(cid:48)\nq, C1 > 0, we have that for\nany standard round t, with a probability at least 1 \u2212 \u03b4t/4, E1(t) holds.\nLemma 5.3. For any \u03c6L, \u03c6M , \u03c6U such that 0 < \u03c6L < \u03c6M < 1 \u2212 \u03c6U < 1, there are suf\ufb01ciently\nlarge constants Cq, C(cid:48)\nq, C1, N\ufb01nal > 0 such that for any standard round t, with a probability at least\n1 \u2212 \u03b4t/4, E2(t) holds.\nLemma 5.4. For any \u03c6L, \u03c6M , \u03c6U such that 0 < \u03c6L < \u03c6M < 1 \u2212 \u03c6U < 1, there is some \u03b8 \u2208 (0, 1)\nsuch that for a suf\ufb01ciently large C2 > 0, in any standard round t, with a probability at least 1 \u2212 \u03b4t/4,\nE1(t) implies E3(t).\n\nt+1 \u2265 w\u2217\n\nt \u2212 \u0001t, and (2) Wt+1 \u2264 (1 \u2212 \u03c6L)Wt or |St+1| \u2264 N\ufb01nal.\n\nThe following lemma shows that these events guarantee that a good arm always remains in the\ncandidate set, and that the reduction of the candidate set in each round is as required.\nLemma 5.5. For any standard round t in which E1, E2, E3 all hold, we have, with a probability at\nleast 1 \u2212 \u03b4t, that (1) w\u2217\nProof. To prove the \ufb01rst part, observe that by E3, St+1 \\ Mt (cid:54)= \u2205. Let j \u2208 argmaxj\u2208St+1\\Mt \u02c6wj,t.\nBy the de\ufb01nition of St+1 in Alg. 1, it includes the arms in St with the largest estimated weights. Thus,\nwe also have j \u2208 argmaxj\u2208St\\Mt \u02c6wj,t. Therefore, since i\u2217\nt \u2208 St \\ Mt, it follows that \u02c6wj,t \u2265 \u02c6w\u2217\nt .\nThus, since j /\u2208 Mt, it follows j is not t-bad, so wj \u2265 w\u2217\nt \u2212 \u0001t. Combined with w\u2217\nt+1 \u2265 wj, this\ncompletes the proof of the \ufb01rst part. For the second part, note that St+1 \u2286 St and Lt \u2286 St. By\nE3, St+1 \u2229 Lt = \u2205. Therefore, Wt+1 + WLt \u2264 Wt. If |St+1| > N\ufb01nal then by E2, WLt \u2265 \u03c6LWt.\nTherefore, Wt+1 \u2264 Wt(1 \u2212 \u03c6L).\n\nCombining the lemmas above, the main result is shown in the following theorem.\nTheorem 5.6. For any setting of the constants except for C3 and \u03c1 that satis\ufb01es the lemmas above,\nthere is setting of C3 > 0 and \u03c1 \u2208 (0, 1) such that with a probability at least 1 \u2212 \u03b4, MAB-PPR\nterminates and returns an \u0001-best arm, and its cost is\n\nO(W \u00b7 log(1/\u03b4) \u00b7 max(\n\n1\n\u00012 ,\n\nV\n\u0001\n\n)).\n\nrounds. By Lemmas 5.1, 5.2, 5.3, and 5.4, and the fact that(cid:80)\u221e\nProof. Condition below on E0 occurring in all rounds, and on E1, E2, E3 occurring in all standard\nt=1 \u03b4t \u2264 \u03b4, this condition holds\nFrom E0(t) it follows that in a round with Wt \u2264 \u0001/8, we have(cid:102)Wt \u2264 \u0001/4, which guarantees that\nwith a probability at least 1 \u2212 \u03b4. By Lemma 5.5, in all standard rounds Wt \u2264 (1 \u2212 \u03c6L)t\u22121W .\nMAB-PPR terminates in line 15. Therefore, MAB-PPR terminates at the latest when t satis\ufb01es\n(1\u2212 \u03c6L)t\u22121W \u2264 \u0001/8. Let T be the last round of the run. By Lemma 5.5, in standard rounds we have\nt+1 \u2265 w\u2217\nw\u2217\n\nt \u2212 \u0001t. Thus,\n\nT \u2265 w\u2217 \u2212 T\u22121(cid:88)\n\nw\u2217\n\nT\u22121(cid:88)\n\nt=1\n\nt=1\n\n6\n\n\u0001t = w\u2217 \u2212 (1 \u2212 \u03c1)\n\n\u03c1t\u22121\u0001/2 \u2265 w\u2217 \u2212 \u0001/2.\n\n(1)\n\n\fConsider the two possible ways of terminating: if the algorithm terminates in line 15, then(cid:102)WT \u2264 \u0001/4.\nBy E0(t), this means that WT \u2264 \u0001/2, thus wr \u2265 w\u2217\nT \u2212 \u0001/2. The other way of terminating is\nwhen |ST| \u2264 N\ufb01nal. In this case, for all i \u2208 ST , \u02c6wi,T is a (q3, (cid:96)(3))-MoM estimate of wi, where\nN\ufb01nal arms, it follows that with a probability at least 1 \u2212 \u03b4T , \u2200i \u2208 ST ,|wi \u2212 \u02c6wi,T| \u2264(cid:113)\n2 log(N\ufb01nal/\u03b4T ) and (cid:96)(3) = C3/\u00012. Set C3 \u2265 96. By Lemma 2.3 and a union bound on\nq3 \u2265 9\n(cid:112)6/(cid:96)(3) \u2264 \u0001/4, where the second inequality follows from Assumption 2.1. The returned arm in this\ni /(cid:96)(3) \u2264\nT \u2212 \u0001/2. Therefore, in both\ntotal cost is at most W(cid:80)T\nT(cid:88)\n\ncase is r \u2208 argmaxi\u2208ST \u02c6wi,T , thus wr \u2265 \u02c6wr,T \u2212 \u0001/4 \u2265 \u02c6w\u2217\ncases we have wr \u2265 w\u2217\nThe cost of any round of MAB-PPR is upper-bounded by Wt(q1(cid:96)t +q2(cid:96)(2) +q+3(cid:96)(3)). Therefore, the\nt=1(1 \u2212 \u03c6L)t\u22121(q1(cid:96)t + q2(cid:96)(2) + q3(cid:96)(3)). We have, for constants C, C(cid:48) > 0\n),\nT\u22121(cid:88)\n2 (1 \u2212 \u03c1)\u03c1t\u22121\u0001. Therefore, the total cost is upper bounded by\nC(cid:48)\n\u03b4t\n\n(that depend on the constants used in Alg. 1), that q1(cid:96)t + q2(cid:96)(2) + q3(cid:96)(3) \u2264 C log( C(cid:48)\nwhere \u03b4t = \u03b4/2t and \u0001t = 1\n(1 \u2212 \u03c6L)t\u22121 \u00b7 C log(\n\n) \u2264 C(cid:48)(cid:48) \u00b7 W log(C(cid:48)(cid:48)(cid:48)/\u03b4) max(\n1\nV\nW\n,\n(\u03c12)t\u22121\n\u00012\n\u0001t\n\u221a\nt\nFor constants C(cid:48)(cid:48), C(cid:48)(cid:48)(cid:48) > 0. Setting \u03c1 \u2208 (\n1 \u2212 \u03c6L, 1), the sum on t is upper-bounded by a constant,\ngiving the required upper bound on the cost.\n\nT \u2212 \u0001/2. Combined with Eq. (1), this proves that wr \u2265 w\u2217 \u2212 \u0001.\n\nT \u2212 \u0001/4 \u2265 w\u2217\n\n) \u00b7 max( 1\n\n1\n\u00012 ,\n\n) max(\n\nt(1 \u2212 \u03c6L)t\u22121\n\n, V\n\u0001t\n\n6\u03c32\n\nV\n\u0001\n\nt=1\n\nt=1\n\n\u00012\nt\n\n\u03b4t\n\n)\n\n,\n\nWe now prove lemmas that the events E1, E2, E3 hold with a high probability. We start with\nLemma 5.2, which shows this for E1(t), which states that WMt < \u03c6U Wt.\nProof of Lemma 5.2. Fix a round t. De\ufb01ne the event E4 := { \u02c6w\u2217\nt \u2265 w\u2217\nq \u2265 8 and C1 \u2265 24, we have q1 \u2265 9\n2 log(8/\u03b4t) and (cid:96)t \u2265 24/\u00012\nC(cid:48)\nt . By Assumption 2.1, \u03c32\ni\u2217\nHence, by Lemma 2.3, with a probability 1 \u2212 \u03b4t/8, |wt \u2212 \u02c6w\u2217\nLet i \u2208 St be a t-bad arm. Let \u02c6wj\n\u02c6wi,t in line 5. Let \u03b11 \u2208 (1 \u2212 \u03c6U\nhold, then since i is t-bad, \u02c6wj\n\ni for j \u2208 [q1] be the j\u2019th mean estimate of wi used to get the estimate\n2 , 1). Set \u03b12 = 1 \u2212 1\u2212\u03c6U /2\n, and set \u03b13 \u2208 (0, \u03b12(1 \u2212 \u03b11)). If E4\nt \u2265 wi + \u0001t/2. Denote Pt[\u00b7] := P[\u00b7 | St]. Then,\n\nt | \u2264(cid:112)6/(cid:96)t \u2264 \u0001t/2, hence E4 holds.\n\nt \u2212 \u0001t/2}. Setting Cq \u2265 9\n2,\n\u2264 1.\n\nt implies \u02c6wj\n\ni \u2265 \u02c6w\u2217\n\ni \u2265 \u02c6w\u2217\n\n\u03b11\n\nt\n\nPt[ \u02c6wj\n\ni \u2265 \u02c6w\u2217\n\ni \u2265 wi + \u0001t/2]/Pt[E4] \u2264 2Pt[ \u02c6wj\nThe last inequality follows from Chebychev\u2019s inequality, since \u03c32\n(cid:96)t \u2265 8\nconclude that\n\nt | E4] \u2264 Pt[ \u02c6wj\ni \u2265 \u02c6w\u2217\n(cid:88)\n\n| E4] \u2264 \u03b13. Denote M j\n\n, hence Pt[ \u02c6wj\n| E4] =\n\nt | E4] \u2264 (cid:88)\n\nwi \u00b7 Pt[i \u2208 M j\n\nEt[WM j\n\n\u03b13\u00012\nt\n\nt\n\nt\n\nt\n\ni \u2265 wi + \u0001t/2] \u2264\ni \u2264 1. Setting C1 \u2265 8\n\n2\n\n(cid:96)t(\u0001t/2)2 .\n\n:= {i | i is t-bad and \u02c6wj\n\n\u03b13\n\ni \u2265 \u02c6w\u2217\nt | E4] \u2264 \u03b13Wt.\n\n, we have\nt }. We\n\nwi \u00b7 Pt[ \u02c6wj\n\ni \u2265 \u02c6w\u2217\n\ni\u2208St\n\u2265 \u03b12Wt | E4] \u2264 \u03b13Wt\n\ni\u2208St\n\n(cid:80)\n\n< 1 \u2212 \u03b11. De\ufb01ne\nq/\u03b4t). Setting\nq \u2265 8, we have by Hoeffding\u2019s inequality and the independence\ni } for j \u2208 [q1] that Pt[E5 | E4] \u2265 1 \u2212 \u03b4t/8. Therefore, combined with Pt[E4] \u2265 1 \u2212 \u03b4t/8,\n\nBy Markov\u2019s inequality, it follows that Pt[WM j\nthe event E5 := { 1\nCq \u2265 1/(2(1\u2212 \u03b11 \u2212 \u03b13/\u03b12)2) and C(cid:48)\nof {M j\nwe get that for any St, Pt[E5] \u2265 1 \u2212 \u03b4/4. Under E5, we have\n\n\u2265 \u03b12Wt] < 1 \u2212 \u03b11}. We have q1 \u2265 Cq log(C(cid:48)\n\nI[WM j\n\n\u03b12Wt\n\nq1\n\nj\n\nt\n\nt\n\n< \u03b12\u03b11Wt + (1 \u2212 \u03b11)Wt = (\u03b12\u03b11 + 1 \u2212 \u03b11)Wt = \u03c6U Wt/2.\n\n(2)\n\nWhere the last equality follows from the de\ufb01nition of \u03b12. On the other hand, since \u02c6wi,t is the median\nof { \u02c6wj\n\ni}j\u2208[q1], and i \u2208 Mt implies \u02c6wi,t \u2265 \u02c6w\u2217\nWMt =\n\nt , we have\nwi \u00b7 I[|{j | i \u2208 M j\n\n(cid:88)\nCombined with Eq. (2), we get that with a probability of 1 \u2212 \u03b4t/4, WMt < \u03c6U Wt.\n\ni\u2208St\nI[i \u2208 M j\n\ni\u2208St\n\u2264 2\nq1\n\nwi \u00b7 I[i \u2208 M j\n\n(cid:88)\n\n(cid:88)\n\nj\u2208[q1]\n\nj\u2208[q1]\n\nt ] =\n\nt ] =\n\n2\nq1\n\n2\nq1\n\ni\u2208St\n\ni\u2208St\n\nwi\n\nj\u2208[q1]\n\nt }| \u2265 q1/2]\n\nWM j\n\nt\n\n.\n\n1\nq1\n\nt\n\nj\n\nWM j\n\n(cid:88)\n(cid:88)\nwi \u00b7 I[i \u2208 Mt] \u2264 (cid:88)\n(cid:88)\n\n(cid:88)\n\n7\n\n\fWe now prove Lemma 5.3, which states that with a probability at least 1 \u2212 \u03b4t/4, E2(t) holds in a\nstandard round, i.e., WLt \u2265 \u03c6LWt or |St \\ Lt| \u2264 N\ufb01nal. In this proof we use the following lemma,\nproved in the supplementary material (Appendix C).\n\nLemma 5.7. Let Y1, . . . , Yz be random variables with means \u00b51, . . . , \u00b5z. Let \u00afY := (cid:80)\n\u00b5 :=(cid:80)\n\ni\u2208[z] Yi,\ni\u2208[z] \u00b5i = E[ \u00afY ]. Let \u03c32 := Var( \u00afY ) < \u221e. For i \u2208 [z], let \u02c6\u00b5i be the (q, (cid:96))-MoM estimate\nof \u00b5i based on q(cid:96) i.i.d. draws of copies of Yi. Then for any \u03b3 > 0, with a probability at least\n1 \u2212 exp(\u2212q/18), |{i \u2208 [z] | \u02c6\u00b5i \u2265 \u03b3\u00b5}| \u2264 4\n\n(cid:113) 6\u03c32\n\n\u03b3 (1 +\n\n\u00b52(cid:96) ).\n\n2 and C(cid:48)\n\nq \u2265 8/\u03b2, so that q1 \u2265 9\n\n2 log((8/\u03b2)/\u03b4t). Recall that \u03c32\n\n(cid:112)6wi/(cid:96)t. Setting C1 \u2265 6/\u03b12, it follows that (cid:96)t \u2265 6/(\u03b12\u0001t). We have \u0001t \u2264 \u0001/2 for all t. In addition,\n\n\u221a\nProof of Lemma 5.3. We will assume that WLt < \u03c6LWt and conclude the required upper bound on\n|St \\ Lt| with a high probability. Let \u03b2 := \u03c6M \u2212 \u03c6L. Let \u03bd \u2208 (0, \u03b2), and set \u03b1 := 1\n2 (\u03b2 \u2212 \u03bd)/\n\u03b2.\nLet j := jkt+1, where kt is given in the de\ufb01nition of Lt and {jl} is the ordering set in line 6 in round\nt. By the de\ufb01nition of Lt, WLt + wj > \u03c6M Wt. By our assumption, WLt < \u03c6LWt. Therefore,\nwj > (\u03c6M \u2212 \u03c6L)Wt = \u03b2Wt. De\ufb01ne J = {i \u2208 St | wi \u2265 \u03b2Wt}. We have |J| \u2264 1/\u03b2.\ni \u2264 wi by Assumption 2.1.\nSet Cq \u2265 9\nTherefore, by Lemma 2.3 and a union bound, with a probability at least 1\u2212 \u03b4t/8, \u2200i \u2208 J,|wi \u2212 \u02c6wi| \u2264\nsince t is a standard round, \u0001/8 \u2264 Wt. Therefore, for i \u2208 J, \u0001t \u2264 4Wt \u2264 4wi/\u03b2. It follows that\n\u221a\nwith a probability at least 1 \u2212 \u03b4t/8, \u2200i \u2208 J,|wi \u2212 \u02c6wi| \u2264 (2\u03b1/\n\u03b2)wi = (1 \u2212 \u03bd/\u03b2)wi, where the\nequality follows from the de\ufb01nition of \u03b1. Hence \u02c6wi \u2265 (\u03bd/\u03b2) \u00b7 wi \u2265 \u03bdWt. In particular, this holds\nfor j = jkt+1 as set above. Condition below on this event.\nBy the de\ufb01nition of Lt and j, for all i \u2208 St \\ Lt, \u02c6wi \u2265 \u02c6wj. Therefore, by the event above, for all\ni \u2208 St \\ Lt, \u02c6wi \u2265 \u03bdWt. Hence, |St \\ Lt| \u2264 |{i \u2208 St | \u02c6wi \u2265 \u03bdWt}|. To bound the RHS, recall\n\u2264 V Wt by Assumption 2.2. Thus, applying Lemma 5.7, we have that by setting Cq \u2265 18,\nthat \u03c32\nSt\n). We have (cid:96)t \u2265 C1V /\u0001t \u2265 C1V /(4Wt).\nw.p. 1 \u2212 \u03b4t/8, |{i \u2208 St | \u02c6wi \u2265 \u03bdWt}| \u2264 4\nTherefore,\n\n(cid:113) 6V\n(1 +(cid:112)24/C1) =: N\ufb01nal.\n\n(3)\nBy a union bound, with a probability 1 \u2212 \u03b4t/4, the event above and Eq. (3) hold simultaneously, in\nwhich case WLt < \u03c6LWt implies |St \\ Lt| \u2264 N\ufb01nal.\n\n|{i \u2208 St | \u02c6wi \u2265 \u03bdWt}| \u2264 4\n\u03bd\n\n\u03bd (1 +\n\nWt(cid:96)t\n\nLastly, we prove Lemma 5.4, which states that in a standard round t, with a probability at least\n1 \u2212 \u03b4t/4, E1(t) implies E3(t), i.e., St+1 \\ Mt (cid:54)= \u2205 and St+1 \u2229 Lt = \u2205.\nProof of Lemma 5.4. Fix a round t. Let Al := {j1, . . . , jl}, where {jl} is the ordering set in line 6,\n\nand let(cid:102)WAl :=(cid:102)WAl,t be the estimate of WAl calculated in line 8 of round t. By the assumption of\n\n\u221a\nthe lemma, \u03c6M < 1 \u2212 \u03c6U . Let R \u2208 (0, 1 \u2212 \u03c6U ) be the solution to \u03c6M +R\n1+R = 0. The\n1\u2212R\nsolution is in this range, since since for R = 0, the LHS is equal to \u03c6M \u2212 (1 \u2212 \u03c6U ) < 0, and for\nR = 1 \u2212 \u03c6U , the LHS is positive. Set \u03b8 := 1\u2212\u03c6U\u2212R\n1+R .\n2 log(12/\u03b4t). By setting C2 \u2265 48/R2, we get (cid:96)(2) \u2265 48V /(R2\u0001). Therefore, by\nRecall that q2 = 9\nLemma 2.3 and Assumption 2.2, for any \ufb01xed l the following holds with a probability 1 \u2212 \u03b4t/12:\n\n\u2212 1\u2212\u03c6U\u2212R\n\n\u03c6M\n\n|(cid:102)WAl \u2212 WAl| \u2264(cid:113)\n\n6V WAl /(cid:96)(2) \u2264(cid:112)R2WAl \u0001/8 \u2264 R(cid:112)WAl Wt.\n\n(4)\nThe last inequality follows since t is a standard round, hence Wt \u2265 \u0001/8. Let nt be the smallest\n\u2265 Wt \u2212 WMt. With a probability 1 \u2212 \u03b4t/4, Eq. (4) holds simultaneously for\ninteger such that WAnt\nl = nt, l = kt (so Al \u2261 Lt) and l = |St| (so Al \u2261 St). Condition below on this combined event. It\nfollows from the de\ufb01nition of Lt that\n\n(cid:102)WLt \u2264 WLt + R(cid:112)WLtWt \u2264 (\u03c6M + R(cid:112)\u03c6M )Wt.\n\nWt > (1 \u2212 \u03c6U )Wt \u2212 RWt = (1 \u2212 \u03c6U \u2212 R)Wt.\n\nWAnt\n\n8\n\nSuppose that E1(t) holds, so WMt < \u03c6U Wt. By de\ufb01nition of nt, WAnt\n\n> (1 \u2212 \u03c6U )Wt. Therefore,\n\n(cid:102)WAnt\n\n\u2265 WAnt\n\n\u2212 R\n\n(cid:113)\n\n\fLastly, from Eq. (4) for l = |St|, we get Wt(1 \u2212 R) \u2264 (cid:102)Wt \u2264 Wt(1 + R). Now, MAB-PPR sets\n/(cid:102)Wt \u2264 \u03b8. From the inequalities\nSt+1 = St \\ ANt, where Nt is the largest number that satis\ufb01es(cid:102)WANt\n= \u03b8. Hence, kt \u2264 Nt < nt.\nabove, we have\nFrom kt \u2264 Nt, we have St+1 \u2229 Lt = \u2205. From Nt < nt, we have St+1 \u2261 St \\ ANt \u2287 St \\ Ant\u22121.\nTherefore Wt+1 \u2265 Wt\u2212WAnt\u22121. By the de\ufb01nition of nt, we have WAnt\u22121 < Wt\u2212WMt. Therefore,\nWt+1 \u2265 WMt. It follows that St+1 \\ Mt (cid:54)= \u2205. Thus, E1(t) implies E3(t).\n\n1+R = \u03b8 and (cid:102)WLt(cid:102)Wt\n\n> 1\u2212\u03c6U\u2212R\n\n(cid:102)WAnt(cid:102)Wt\n\n\u221a\n\u2264 \u03c6M +R\n1\u2212R\n\n\u03c6M\n\nHaving proved Lemma 5.2,Lemma 5.3, and Lemma 5.4, this \ufb01nalizes the proof of Theorem 5.6.\n\n6 Conclusion\n\nIn this work we showed that it is possible to identify an \u0001-best arm in a pay-per-reward multi-armed-\nbandit setting at a cost that does not depend on the number of arms, and depends only linearly on the\ntotal expected reward. In future work, we plan to study instance-speci\ufb01c bounds for this setting.\n\nAcknowledgements\n\nSivan Sabato was supported in part by the Israel Science Foundation (grant No. 555/15).\n\nReferences\n[1] D. Agarwal, B.-C. Chen, and P. Elango. Explore/exploit schemes for web content optimization.\n\nIn 2009 Ninth IEEE International Conference on Data Mining, pages 1\u201310. IEEE, 2009.\n\n[2] N. Alon, Y. Matias, and M. Szegedy. The space complexity of approximating the frequency\n\nmoments. Journal of Computer and system sciences, 58(1):137\u2013147, 1999.\n\n[3] J.-Y. Audibert and S. Bubeck. Best arm identi\ufb01cation in multi-armed bandits. In COLT-23th\n\nConference on learning theory-2010, pages 13\u2013p, 2010.\n\n[4] M. Aziz, J. Anderton, E. Kaufmann, and J. Aslam. Pure exploration in in\ufb01nitely-armed bandit\n\nmodels with \ufb01xed-con\ufb01dence. arXiv preprint arXiv:1803.04665, 2018.\n\n[5] J. S. Banks and R. K. Sundaram. Denumerable-armed bandits. Econometrica: Journal of the\n\nEconometric Society, pages 1071\u20131096, 1992.\n\n[6] D. A. Berry, R. W. Chen, A. Zame, D. C. Heath, and L. A. Shepp. Bandit problems with\n\nin\ufb01nitely many arms. The Annals of Statistics, 25(5):2103\u20132116, 1997.\n\n[7] C. Brownlees, E. Joly, and G. Lugosi. Empirical risk minimization for heavy-tailed losses. The\n\nAnnals of Statistics, 43(6):2507\u20132536, 2015.\n\n[8] S. Bubeck, R. Munos, and G. Stoltz. Pure exploration in multi-armed bandits problems. In\n\nInternational conference on Algorithmic learning theory, pages 23\u201337. Springer, 2009.\n\n[9] S. Bubeck, N. Cesa-Bianchi, and G. Lugosi. Bandits with heavy tail. IEEE Transactions on\nInformation Theory, 59(11):7711\u20137717, Nov 2013. ISSN 0018-9448. doi: 10.1109/TIT.2013.\n2277869.\n\n[10] A. Carpentier and A. Locatelli. Tight (lower) bounds for the \ufb01xed budget best arm identi\ufb01cation\n\nbandit problem. In Conference on Learning Theory, pages 590\u2013604, 2016.\n\n[11] A. Carpentier and M. Valko. Simple regret for in\ufb01nitely many armed bandits. In International\n\nConference on Machine Learning, pages 1133\u20131141, 2015.\n\n[12] L. Chen, J. Li, and M. Qiao. Towards instance optimal bounds for best arm identi\ufb01cation.\nIn S. Kale and O. Shamir, editors, Proceedings of the 2017 Conference on Learning The-\nory, volume 65 of Proceedings of Machine Learning Research, pages 535\u2013592, Amsterdam,\nNetherlands, 07\u201310 Jul 2017. PMLR.\n\n9\n\n\f[13] W. Ding, T. Qin, X.-D. Zhang, and T.-Y. Liu. Multi-armed bandit with budget constraint and\n\nvariable costs. In Twenty-Seventh AAAI Conference on Arti\ufb01cial Intelligence, 2013.\n\n[14] E. Even-Dar, S. Mannor, and Y. Mansour. Pac bounds for multi-armed bandit and markov\ndecision processes. In International Conference on Computational Learning Theory, pages\n255\u2013270. Springer, 2002.\n\n[15] E. Even-Dar, S. Mannor, and Y. Mansour. Action elimination and stopping conditions for the\nmulti-armed bandit and reinforcement learning problems. Journal of machine learning research,\n7(Jun):1079\u20131105, 2006.\n\n[16] D. Hsu and S. Sabato. Loss minimization and parameter estimation with heavy tails. Journal of\n\nMachine Learning Research, 17(18):1\u201340, 2016.\n\n[17] K. G. Jamieson and R. D. Nowak. Best-arm identi\ufb01cation algorithms for multi-armed bandits\nin the \ufb01xed con\ufb01dence setting. 2014 48th Annual Conference on Information Sciences and\nSystems (CISS), pages 1\u20136, 2014.\n\n[18] E. Joly, G. Lugosi, and R. I. Oliveira. On the estimation of the mean of a random vector.\n\nElectronic Journal of Statistics, 11(1):440\u2013451, 2017.\n\n[19] Z. Karnin, T. Koren, and O. Somekh. Almost optimal exploration in multi-armed bandits. In\n\nInternational Conference on Machine Learning, pages 1238\u20131246, 2013.\n\n[20] H. Li and Y. Xia. In\ufb01nitely many-armed bandits with budget constraints. In Thirty-First AAAI\n\nConference on Arti\ufb01cial Intelligence, 2017.\n\n[21] G. Lugosi and S. Mendelson. Risk minimization by median-of-means tournaments. arXiv\n\npreprint arXiv:1608.00757, 2016.\n\n[22] S. Mannor and J. N. Tsitsiklis. The sample complexity of exploration in the multi-armed bandit\n\nproblem. Journal of Machine Learning Research, 5(Jun):623\u2013648, 2004.\n\n[23] S. Minsker. Geometric median and robust estimation in banach spaces. Bernoulli, 21(4):2308\u2013\n\n2335, 11 2015. doi: 10.3150/14-BEJ645. URL https://doi.org/10.3150/14-BEJ645.\n\n[24] S. Pandey, D. Agarwal, D. Chakrabarti, and V. Josifovski. Bandits for taxonomies: A model-\nbased approach. In Proceedings of the 2007 SIAM International Conference on Data Mining,\npages 216\u2013227. SIAM, 2007.\n\n[25] L. Tran-Thanh, A. Chapman, E. M. de Cote, A. Rogers, and N. R. Jennings. Epsilon\u2013\ufb01rst\npolicies for budget\u2013limited multi-armed bandits. In Twenty-Fourth AAAI Conference on Arti\ufb01cial\nIntelligence, 2010.\n\n[26] L. Tran-Thanh, A. Chapman, A. Rogers, and N. R. Jennings. Knapsack based optimal poli-\ncies for budget\u2013limited multi\u2013armed bandits. In Twenty-Sixth AAAI Conference on Arti\ufb01cial\nIntelligence, 2012.\n\n[27] H. P. Vanchinathan, A. Marfurt, C.-A. Robelin, D. Kossmann, and A. Krause. Discovering\nvaluable items from massive data. In Proceedings of the 21th ACM SIGKDD International\nConference on Knowledge Discovery and Data Mining, pages 1195\u20131204. ACM, 2015.\n\n[28] Y. Wang, J.-Y. Audibert, and R. Munos. Algorithms for in\ufb01nitely many-armed bandits. In\n\nAdvances in Neural Information Processing Systems, pages 1729\u20131736, 2009.\n\n[29] Y. Xia, T. Qin, W. Ma, N. Yu, and T.-Y. Liu. Budgeted multi-armed bandits with multiple plays.\n\nIn IJCAI, pages 2210\u20132216, 2016.\n\n[30] Y. Xia, T. Qin, N. Yu, and T.-Y. Liu. Best action selection in a stochastic environment. In\nProceedings of the 2016 International Conference on Autonomous Agents & Multiagent Systems,\npages 758\u2013766. International Foundation for Autonomous Agents and Multiagent Systems,\n2016.\n\n10\n\n\f[31] X. Yu, H. Shao, M. R. Lyu, and I. King. Pure exploration of multi-armed bandits with heavy-\ntailed payoffs. In Proceedings of the Thirty-Fourth Conference on Uncertainty in Arti\ufb01cial\nIntelligence, pages 937\u2013946, 2018.\n\n[32] Q. Zhou, X. Zhang, J. Xu, and B. Liang. Large-scale bandit approaches for recommender\nsystems. In Neural Information Processing, pages 811\u2013821, Cham, 2017. Springer International\nPublishing.\n\n11\n\n\f", "award": [], "sourceid": 1663, "authors": [{"given_name": "Sivan", "family_name": "Sabato", "institution": "Ben-Gurion University of the Negev"}]}