{"title": "Refined Lower Bounds for Adversarial Bandits", "book": "Advances in Neural Information Processing Systems", "page_first": 1198, "page_last": 1206, "abstract": "We provide new lower bounds on the regret that must be suffered by adversarial bandit algorithms. The new results show that recent upper bounds that either (a) hold with high-probability or (b) depend on the total loss of the best arm or (c) depend on the quadratic variation of the losses, are close to tight. Besides this we prove two impossibility results. First, the existence of a single arm that is optimal in every round cannot improve the regret in the worst case. Second, the regret cannot scale with the effective range of the losses. In contrast, both results are possible in the full-information setting.", "full_text": "Re\ufb01ned Lower Bounds for Adversarial Bandits\n\nS\u00e9bastien Gerchinovitz\n\nInstitut de Math\u00e9matiques de Toulouse\n\nUniversit\u00e9 Toulouse 3 Paul Sabatier\n\nToulouse, 31062, France\n\nTor Lattimore\n\nDepartment of Computing Science\n\nUniversity of Alberta\nEdmonton, Canada\n\nsebastien.gerchinovitz@math.univ-toulouse.fr\n\ntor.lattimore@gmail.com\n\nAbstract\n\nWe provide new lower bounds on the regret that must be suffered by adversarial\nbandit algorithms. The new results show that recent upper bounds that either (a)\nhold with high-probability or (b) depend on the total loss of the best arm or (c)\ndepend on the quadratic variation of the losses, are close to tight. Besides this we\nprove two impossibility results. First, the existence of a single arm that is optimal\nin every round cannot improve the regret in the worst case. Second, the regret\ncannot scale with the effective range of the losses. In contrast, both results are\npossible in the full-information setting.\n\n1\n\nIntroduction\n\nT(cid:88)\n\nt=1\n\nT(cid:88)\n\nWe consider the standard K-armed adversarial bandit problem, which is a game played over T\nrounds between a learner and an adversary. In every round t \u2208 {1, . . . , T} the learner chooses a\nprobability distribution pt = (pi,t)1(cid:54)i(cid:54)K over {1, . . . , K}. The adversary then chooses a loss vector\n(cid:96)t = ((cid:96)i,t)1(cid:54)i(cid:54)K \u2208 [0, 1]K, which may depend on pt. Finally the learner samples an action from pt\ndenoted by It \u2208 {1, . . . , K} and observes her own loss (cid:96)It,t. The learner would like to minimise her\nregret, which is the difference between cumulative loss suffered and the loss suffered by the optimal\naction in hindsight:\n\nRT ((cid:96)1:T ) =\n\n(cid:96)It,t \u2212 min\n1(cid:54)i(cid:54)K\n\n(cid:96)i,t ,\n\nt=1\n\nExp3, which satis\ufb01es E[RT ((cid:96)1:T )] = O((cid:112)KT log(K))) where the expectation is taken over the\n\nwhere (cid:96)1:T \u2208 [0, 1]T K is the sequence of losses chosen by the adversary. A famous strategy is called\n\nrandomness in the algorithm and the choices of the adversary [Auer et al., 2002]. There is also a\n\u221a\nlower bound showing that for every learner there is an adversary for which the expected regret is\nE[RT ((cid:96)1:T )] = \u2126(\n\u221a\nKT ) [Auer et al., 1995]. If the losses are chosen ahead of time, then the adver-\nsary is called oblivious, and in this case there exists a learner for which E[RT ((cid:96)1:T )] = O(\nKT )\n[Audibert and Bubeck, 2009]. One might think that this is the end of the story, but it is not so. While\nthe worst-case expected regret is one quantity of interest, there are many situations where a re\ufb01ned\nregret guarantee is more informative. Recent research on adversarial bandits has primarily focussed\non these issues, especially the questions of obtaining regret guarantees that hold with high probability\nas well as stronger guarantees when the losses are \u201cnice\u201d in some sense. While there are now a wide\nrange of strategies with upper bounds that depend on various quantities, the literature is missing lower\nbounds for many cases, some of which we now provide.\nWe focus on three classes of lower bound, which are described in detail below. The \ufb01rst addresses the\noptimal regret achievable with high probability, where we show there is little room for improvement\nover existing strategies. Our other results concern lower bounds that depend on some kind of regularity\nin the losses (\u201cnice\u201d data). Speci\ufb01cally we prove lower bounds that replace T in the regret bound\nwith the loss of the best action (called \ufb01rst-order bounds) and also with the quadratic variation of the\nlosses (called second-order bounds).\n\n30th Conference on Neural Information Processing Systems (NIPS 2016), Barcelona, Spain.\n\n\f(1)\n\n(2)\n\nHigh-probability bounds Existing strategies Exp3.P [Auer et al., 2002] and Exp3-IX [Neu, 2015a]\nare tuned with a con\ufb01dence parameter \u03b4 \u2208 (0, 1) and satisfy, for all (cid:96)1:T \u2208 [0, 1]KT ,\n\nP(cid:16)\n(cid:32)\n\n(cid:17) (cid:54) \u03b4\nRT ((cid:96)1:T ) (cid:62) c(cid:112)KT log(K/\u03b4)\n(cid:32)(cid:112)log(K) +\n\n\u221a\nRT ((cid:96)1:T ) (cid:62) c\n\nKT\n\nfor some universal constant c > 0. An alternative tuning of Exp-IX or Exp3.P [Bubeck and Cesa-\nBianchi, 2012] leads to a single algorithm for which, for all (cid:96)1:T \u2208 [0, 1]KT ,\nlog(1/\u03b4)\n\n(cid:33)(cid:33)\n\n\u2200\u03b4 \u2208 (0, 1)\n\nP\n\n(cid:112)log(K)\n\n(cid:54) \u03b4 .\n\nThe difference is that in (1) the algorithm depends on \u03b4 while in (2) it does not. The cost of not\nknowing \u03b4 is that the log(1/\u03b4) moves outside the square root. In Section 2 we prove two lower\nbounds showing that there is little room for improvement in either (1) or (2).\nFirst-order bounds An improvement over the worst-case regret bound of O(\nT K) is the\nso-called improvement for small losses. Speci\ufb01cally, there exist strategies (eg., FPL-TRIX by Neu\n[2015b] with earlier results by Stoltz [2005], Allenberg et al. [2006], Rakhlin and Sridharan [2013])\nsuch that for all (cid:96)1:T \u2208 [0, 1]KT\n\n\u221a\n\nE[RT ((cid:96)1:T )] (cid:54) O\n\nL\u2217\nT K log(K) + K log(KT )\n\n, with L\u2217\n\nT = min\n1(cid:54)i(cid:54)K\n\n(cid:96)i,t ,\n\n(3)\n\n(cid:18)(cid:113)\n\n(cid:19)\n\nT(cid:88)\n\nt=1\n\n\u221a\nwhere the expectation is with respect to the internal randomisation of the algorithm (the losses are\n\ufb01xed). This result improves on the O(\n(cid:54) T is always guaranteed and\nsometimes L\u2217\nT is much smaller than T . In order to evaluate the optimality of this bound, we \ufb01rst\nrewrite it in terms of the small-loss balls B\u03b1,T de\ufb01ned for all \u03b1 \u2208 [0, 1] and T (cid:62) 1 by\n\nKT ) bounds since L\u2217\n\nT\n\nB\u03b1,T (cid:44)\n\n(cid:27)\n\n(cid:96)1:T \u2208 [0, 1]KT :\n\n(cid:26)\nE[RT ((cid:96)1:T )] (cid:54) O(cid:16)(cid:112)\u03b1T K log(K) + K log(KT )\n\nL\u2217\nT\nT\n\n(cid:54) \u03b1\n\n.\n\n(4)\n\n(cid:17)\n\n.\n\nCorollary 1. The \ufb01rst-order regret bound (3) of Neu [2015b] is equivalent to:\n\n\u2200\u03b1 \u2208 [0, 1],\n\nsup\n\n(cid:96)1:T \u2208B\u03b1,T\n\n\u221a\nThe proof is straightforward. Our main contribution in Section 3 is a lower bound of the order of\n\u03b1T K for all \u03b1 \u2208 \u2126(log(T )/T ). This minimax lower bound shows that we cannot hope for a better\nbound than (3) (up to log factors) if we only know the value of L\u2217\nT .\n\nSecond-order bounds Another type of improved regret bound was derived by Hazan and Kale\n[2011b] and involves a second-order quantity called the quadratic variation:\n\n(cid:80)T\n\nQT =\n\n(cid:107)(cid:96)t \u2212 \u00b5T(cid:107)2\n\n2\n\n(cid:54) T K\n4\n\n,\n\n(5)\n\nwhere \u00b5T = 1\nt=1 (cid:96)t is the mean of all loss vectors. (In other words, QT /T is the sum of the\nT\nempirical variances of all the K arms). Hazan and Kale [2011b] addressed the general online linear\noptimisation setting. In the particular case of adversarial K-armed bandits with an oblivious adversary\n(as is the case here), they showed that there exists an ef\ufb01cient algorithm such that for some absolute\nconstant c > 0 and for all T (cid:62) 2\n\nK 2(cid:112)QT log T + K 1.5 log2 T + K 2.5 log T\n\n(6)\nAs before we can rewrite the regret bound (6) in terms of the small-variation balls V\u03b1,T de\ufb01ned for\nall \u03b1 \u2208 [0, 1/4] and T (cid:62) 1 by\n\n.\n\n\u2200(cid:96)1:T \u2208 [0, 1]KT ,\n\nE[RT ((cid:96)1:T )] (cid:54) c\n\n(cid:16)\n\n(cid:17)\n\nV\u03b1,T (cid:44)\n\n(cid:96)1:T \u2208 [0, 1]KT :\n\nQT\nT K\n\n(cid:54) \u03b1\n\n.\n\nCorollary 2. The second-order regret bound (6) of Hazan and Kale [2011b] is equivalent to:\n\u2200\u03b1 \u2208 [0, 1/4],\n\nK 2(cid:112)\u03b1T K log T + K 3/2 log2 T + K 5/2 log T\n\nE[RT ((cid:96)1:T )] (cid:54) c\n\nsup\n\n(cid:16)\n\n(7)\n\n(cid:17)\n\n.\n\n(cid:26)\n\n(cid:27)\n\n(cid:96)1:T \u2208V\u03b1,T\n\nT(cid:88)\n\nt=1\n\n2\n\n\fThe proof is straightforward because the losses are deterministic and \ufb01xed in advance by an oblivious\nadversary. In Section 4 we provide a lower bound of order\n\u03b1T K that holds whenever \u03b1 =\n\u2126(log(T )/T ). This minimax lower bound shows that we cannot hope for a bound better than (7) by\nlog T if we only know the value of QT . Closing the gap is left as an open\n\nmore than a factor of K 2\u221a\n\n\u221a\n\nquestion.\n\nTwo impossibility results in the bandit setting We also show in Section 4 that, in contrast to\nthe full-information setting, regret bounds involving the cumulative variance of the algorithm as in\n[Cesa-Bianchi et al., 2007] cannot be obtained in the bandit setting. More precisely, we prove that\ntwo consequences that hold true in the full-information case, namely: (i) a regret bound proportional\nto the effective range of the losses and (ii) a bounded regret if one arm performs best at all rounds,\nmust fail in the worst case for every bandit algorithm.\n\nAdditional notation and key tools Before the theorems we develop some additional notation and\ndescribe the generic ideas in the proofs. For 1 (cid:54) i (cid:54) K let Ni(t) be the number of times action i\nhas been chosen after round t. All our lower bounds are derived by analysing the regret incurred\nby strategies when facing randomised adversaries that choose the losses for all actions from the\nsame joint distribution in every round (sometimes independently for each action and sometimes not).\nBer(\u03b1) denotes the Bernoulli distribution with parameter \u03b1 \u2208 [0, 1]. If P and Q are measures on the\nsame probability space, then KL(P, Q) is the KL-divergence between them. For a < b we de\ufb01ne\nclip[a,b](x) = min{b, max{a, x}} and for x, y \u2208 R we let x \u2228 y = max{x, y}. Our main tools\nthroughout the analysis are the following information-theoretic lemmas. The \ufb01rst bounds the KL\ndivergence between the laws of the observed losses/actions for two distributions on the losses.\nLemma 1. Fix a randomised bandit algorithm and two probability distributions Q1 and Q2\non [0, 1]K. Assume the loss vectors (cid:96)1, . . . , (cid:96)T \u2208 [0, 1]K are drawn i.i.d. from either Q1 or Q2,\nand denote by Qj the joint probability distribution on all sources of randomness when Qj is used\n(formally, Qj = Pint \u2297 (Q\u2297T\n), where Pint is the probability distribution used by the algorithm for\nits internal randomisation). Let t (cid:62) 1. Denote by ht = (Is, (cid:96)Is,s)1(cid:54)s(cid:54)t\u22121 the history available\nat the beginning of round t, by Q(ht,It)\nthe law of (ht, It) under Qj, and by Qj,i the ith marginal\ndistribution of Qj. Then,\n\nj\n\nj\n\n(cid:2)Ni(t \u2212 1)(cid:3) KL(cid:0)Q1,i, Q2,i\n\n(cid:1) .\n\nEQ1\n\n(cid:16)Q(ht,It)\n\n1\n\nKL\n\n(cid:17)\n\nK(cid:88)\n\ni=1\n\n, Q(ht,It)\n\n2\n\n=\n\nResults of roughly this form are well known and the proof follows immediately from the chain rule\nfor the relative entropy and the independence of the loss vectors across time (see [Auer et al., 2002]\nor the supplementary material). One difference is that the losses need not be independent across the\narms, which we heavily exploit in our proofs by using correlated losses. The second key lemma is an\nalternative to Pinsker\u2019s inequality that proves useful when the Kullback-Leibler divergence is larger\nthan 2. It has previously been used for bandit lower bounds (in the stochastic setting) by Bubeck et al.\n[2013].\nLemma 2 (Lemma 2.6 in Tsybakov 2008). Let P and Q be two probability distributions on the\nsame measurable space. Then, for every measurable subset A (whose complement we denote by Ac),\n\nexp(cid:0)\u2212 KL(P, Q)(cid:1) .\n\nP (A) + Q(Ac) (cid:62) 1\n2\n\n2 Zero-Order High Probability Lower Bounds\n\nshows that no strategy can enjoy smaller regret than \u2126((cid:112)KT log(1/\u03b4)) with probability at least\n\nWe prove two new high-probability lower bounds on the regret of any bandit algorithm. The \ufb01rst\n1 \u2212 \u03b4. Upper bounds of this form have been shown for various algorithms including Exp3.P [Auer\net al., 2002] and Exp3-IX [Neu, 2015a]. Although this result is not very surprising, we are not aware\nof any existing work on this problem and the proof is less straightforward than one might expect.\nAn added bene\ufb01t of our result is that the loss sequences producing large regret have two special\nproperties. First, the optimal arm is the same in every round and second the range of the losses in\n\neach round is O((cid:112)K log(1/\u03b4)/T ). These properties will be useful in subsequent analysis.\n\n\u221a\nIn the second lower bound we show that any algorithm for which E[RT ((cid:96)1:T )] = O(\nnecessarily suffer a high probability regret of at least \u2126(\n\nKT ) must\nKT log(1/\u03b4)) for some sequence (cid:96)1:T .\n\n\u221a\n\n3\n\n\fThe important difference relative to the previous result is that strategies with log(1/\u03b4) appearing\ninside the square root depend on a speci\ufb01c value of \u03b4, which must be known in advance.\nTheorem 1. Suppose K (cid:62) 2 and \u03b4 \u2208 (0, 1/4) and T (cid:62) 32(K \u2212 1) log(2/\u03b4), then there exists a\nsequence of losses (cid:96)1:T \u2208 [0, 1]KT such that\n\n(cid:112)(K \u2212 1)T log(1/(4\u03b4))\n\n(cid:19)\n\n(cid:62) \u03b4/2 ,\n\nP\n\nRT ((cid:96)1:T ) (cid:62) 1\n27\n\nwhere the probability is taken with respect to the randomness in the algorithm. Furthermore (cid:96)1:T\ncan be chosen in such a way that there exists an i such that for all t it holds that (cid:96)i,t = minj (cid:96)j,t and\n\nmaxj,k{(cid:96)j,t \u2212 (cid:96)k,t} (cid:54)(cid:112)(K \u2212 1) log(1/(4\u03b4))/T /(4\nfor any (cid:96)1:T \u2208 [0, 1]KT it holds that E[RT ((cid:96)1:T )] (cid:54) C(cid:112)(K \u2212 1)T . Let \u03b4 \u2208 (0, 1/4) satisfy\n(cid:112)(K \u2212 1)/T log(1/(4\u03b4)) (cid:54) C and T (cid:62) 32 log(2/\u03b4). Then there exists (cid:96)1:T \u2208 [0, 1]KT for which\n\nTheorem 2. Suppose K (cid:62) 2, T (cid:62) 1, and there exists a strategy and constant C > 0 such that\n\nlog 2).\n\n\u221a\n\n(cid:18)\n\n(cid:32)\n\n(cid:112)(K \u2212 1)T log(1/(4\u03b4))\n\n(cid:33)\n\n203C\n\n(cid:62) \u03b4/2 ,\n\nP\n\nRT ((cid:96)1:T ) (cid:62)\n\nwhere the probability is taken with respect to the randomness in the algorithm.\nCorollary 3.\n\u2208 [0, 1]KT and \u03b4 \u2208 (0, 1) the regret\nthat\n\nIf p \u2208 (0, 1) and C > 0,\n\nthen there does not exist a strategy such\nis bounded by\n\nfor all T , K,\n\nRT ((cid:96)1:T ) (cid:62) C(cid:112)(K \u2212 1)T logp(1/\u03b4)\n\n(cid:17) (cid:54) \u03b4.\n\nP(cid:16)\n\n(cid:96)1:T\n\nThe corollary follows easily by integrating the assumed high-probability bound and applying Theo-\nrem 2 for suf\ufb01ciently large T and small \u03b4. The proof may be found in the supplementary material.\n\nProof of Theorems 1 and 2 Both proofs rely on a carefully selected choice of correlated stochastic\nlosses described below. Let Z1, Z2, . . . , ZT be a sequence of i.i.d. Gaussian random variables with\nmean 1/2 and variance \u03c32 = 1/(32 log(2)). Let \u2206 \u2208 [0, 1/30] be a constant that will be chosen\ndifferently in each proof and de\ufb01ne K random loss sequences (cid:96)1\n\n1:T , . . . , (cid:96)K\n\n1:T where\n\n\uf8f1\uf8f4\uf8f2\uf8f4\uf8f3clip[0,1](Zt \u2212 \u2206)\n\nclip[0,1](Zt \u2212 2\u2206)\nclip[0,1](Zt)\n\n(cid:96)j\ni,t =\n\nif i = 1\nif i = j (cid:54)= 1\notherwise .\n\n(cid:0)RT ((cid:96)i\n\nFor 1 (cid:54) j (cid:54) K let Qj be the measure on (cid:96)1:T \u2208 [0, 1]KT and I1, . . . , IT when (cid:96)i,t = (cid:96)j\ni,t for all\n1 (cid:54) i (cid:54) K and 1 (cid:54) t (cid:54) T . Informally, Qj is the measure on the sequence of loss vectors and actions\nwhen the learner interacts with the losses sampled from the jth environment de\ufb01ned above.\nLemma 3. Let \u03b4 \u2208 (0, 1) and suppose \u2206 (cid:54) 1/30 and T (cid:62) 32 log(2/\u03b4).\nThen\n1:T )] (cid:62) 7\u2206EQi[T \u2212 Ni(T )]/8.\nQi\nThe proof follows by substituting the de\ufb01nition of the losses and applying Azuma\u2019s inequality to\nshow that clipping does not occur too often. See the supplementary material for details.\n\n1:T ) (cid:62) \u2206T /4(cid:1) (cid:62) Qi (Ni(T ) (cid:54) T /2)\u2212 \u03b4/2 and EQi[RT ((cid:96)i\n\n(cid:112)\u03c32(K \u2212 1) log(1/(4\u03b4))/(2T ) (cid:54) 1/30. By the pigeonhole principle there exists an i > 1 for\n\nProof of Theorem 1. First we choose the value of \u2206 that determines the gaps in the losses by \u2206 =\nwhich EQ1 [Ni(T )] (cid:54) T /(K \u2212 1). Therefore by Lemmas 2 and 1, and the fact that the KL divergence\nbetween clipped Gaussian distributions is always smaller than without clipping,\n\nQ1 (N1(T ) (cid:54) T /2) + Qi (N1(T ) > T /2) (cid:62) 1\n2\n\n\u2212EQ1[Ni(T )](2\u2206)2\n\n(cid:18)\n1:T ) (cid:62) T \u2206/4(cid:1) (cid:62) max{Q1 (N1(T ) (cid:54) T /2) , Qi (Ni(T ) (cid:54) T /2)} \u2212 \u03b4/2\n\n\u2212 2T \u22062\n\u03c32(K \u2212 1)\n\n(cid:0)RT ((cid:96)k\n\n, Q(hT ,IT )\n\n(cid:62) 1\n2\n\n(cid:19)\n\n(cid:18)\n\n= 2\u03b4 .\n\n2\u03c32\n\nexp\n\nexp\n\nexp\n\ni\n\n(cid:62) 1\n2\nBut by Lemma 3\n\nQk\n\nmax\nk\u2208{1,i}\n\n(cid:16)\u2212 KL\n\n(cid:16)Q(hT ,IT )\n(cid:19)\n\n1\n\n(cid:17)(cid:17)\n\n(cid:62) 1\n2\n\n(Q1 (N1(T ) (cid:54) T /2) + Qi (N1(T ) > T /2)) \u2212 \u03b4/2 (cid:62) \u03b4/2 .\n\n4\n\n\fThe result is completed by substituting the value of \u03c32 = 1/(32 log(2)) and by noting that\n\n32\n\n(cid:115)\n\nlog\n\n4\u03b4\n\nQi\n\n(cid:32)\n\nRT ((cid:96)i\n\n= Qi\n\n1:T ) (cid:62)\n\n(cid:18) 1\n\n\u03c32T (K \u2212 1)\n\nTherefore there exists an i \u2208 {1, . . . , K} such that\n\n(cid:19)(cid:33)\nmaxj,k{(cid:96)j,t \u2212 (cid:96)k,t} (cid:54) 2\u2206 (cid:54)(cid:112)(K \u2212 1) log(1/(4\u03b4))/T /(4\n(cid:18) 1\n(cid:35)\n\nProof of Theorem 2. By the assumption on \u03b4 we have \u2206 = 7\u03c32\n16C\nfor all i > 1 that\n\nC(cid:112)(K \u2212 1)T (cid:62) EQ1[RT ((cid:96)1\n\nEQ1[Ni(T )] >\n\n(cid:34) K(cid:88)\n\n\u03c32\n2\u22062 log\n\nNi(T )\n\n>\n\n\u221a\n\n1:T )] (cid:62) 7\u2206\n8\n\nEQ1\n\n.\n\n4\u03b4\n\n1:T ) (cid:62) T \u2206/4(cid:1) (cid:62) \u03b4/2 .\n(cid:0)RT ((cid:96)i\n(cid:113) K\u22121\nlog(cid:0) 1\n(cid:19)\n\nlog 2) Qi-almost surely.\n\n4\u03b4\n\nT\n\n(cid:1) (cid:54) 1/30. Suppose\n\n(8)\n\n7\u03c32(K \u2212 1)\n\n16\u2206\n\nlog\n\n1\n4\u03b4\n\n= C(cid:112)(K \u2212 1)T ,\n1:T ) (cid:62) T \u2206/4(cid:1) (cid:62) \u03b4/2 .\n\n(cid:0)RT ((cid:96)k\n\nThen by the assumption in the theorem statement and the second part of Lemma 3 we have\n\nwhich is a contradiction. Therefore there exists an i > 1 for which Eq. (8) does not hold. Then by the\nsame argument as the previous proof it follows that\n\n(cid:18)\n\ni=2\n\n(cid:112)(K \u2212 1)T log\n\n(cid:19)\n\nQk\n\nmax\nk\u2208{1,i}\n\nRT ((cid:96)k\n\n1:T ) (cid:62) 7\u03c32\n4 \u00b7 16C\n\n1\n4\u03b4\n\n= max\nk\u2208{1,i}\n\nQk\n\nThe result is completed by substituting the value of \u03c32 = 1/(32 log(2)).\nRemark 1. It is possible to derive similar high-probability regret bounds with non-correlated losses.\nHowever the correlation makes the results cleaner (we do not need an additional concentration\nargument to locate the optimal arm) and it is key to derive Corollaries 4 and 5 in Section 4.\n\n3 First-Order Lower Bound\n\nsmall-loss balls B\u03b1,T de\ufb01ned in (4). Theorem 3 below provides a new lower bound of order(cid:112)L\u2217\n\nFirst-order upper bounds provide improvement over minimax bounds when the loss of the optimal\naction is small. Recall from Corollary 1 that \ufb01rst-order bounds can be rewritten in terms of the\nT K,\nwhich matches the best existing upper bounds up to logarithmic factors. As is standard for minimax\nresults this does not imply a lower bound on every loss sequence (cid:96)1:T . Instead it shows that we cannot\nhope for a better bound if we only know the value of L\u2217\nT .\nTheorem 3. Let K (cid:62) 2, T (cid:62) K \u2228 118, and \u03b1 \u2208 [(c log(32T ) \u2228 (K/2))/T, 1/2], where c = 64/9.\nThen for any randomised bandit algorithm sup(cid:96)1:T \u2208B\u03b1,T\n\u03b1T K/27, where the\nexpectation is taken with respect to the internal randomisation of the algorithm.\n\nE[RT ((cid:96)1:T )] (cid:62) \u221a\n\n\u221a\n\nOur proof is inspired by that of Auer et al. [2002, Theorem 5.1]. The key difference is that we take\nBernoulli distributions with parameter close to \u03b1 instead of 1/2. This way the best cumulative loss\nL\u2217\nT is ensured to be concentrated around \u03b1T , and the regret lower bound\ncan be seen to involve the variance \u03b1(1 \u2212 \u03b1)T of the binomial distribution with parameters \u03b1 and T .\nFirst we state the stochastic construction of the losses and prove a general lemma that allows us\nto prove Theorem 3 and will also be useful in Section 4 to a derive a lower bound in terms of the\nquadratic variation. Let \u03b5 \u2208 [0, 1 \u2212 \u03b1] be \ufb01xed and de\ufb01ne K probability distributions (Qj)K\nj=1 on\n[0, 1]KT such that under Qj the following hold:\n\n\u03b1T K \u2248(cid:112)\u03b1(1 \u2212 \u03b1)T K\n\n\u03b1 if i = j.\n\n\u2022 All random losses (cid:96)i,t for 1 (cid:54) i (cid:54) K and 1 (cid:54) t (cid:54) T are independent.\n\u2022 (cid:96)i,t is sampled from a Bernoulli distribution with parameter \u03b1 + \u03b5 if i (cid:54)= j, or with parameter\nLemma 4. Let \u03b1 \u2208 (0, 1), K (cid:62) 2, and T (cid:62) K/(4(1\u2212\u03b1)). Consider the probability distributions Qj\nQj. Then for\n\non [0, 1]KT de\ufb01ned above with \u03b5 = (1/2)(cid:112)\u03b1(1 \u2212 \u03b1)K/T , and set \u00afQ = 1\nany randomised bandit algorithm E[RT ((cid:96)1:T )] (cid:62)(cid:112)\u03b1(1 \u2212 \u03b1)T K/8, where the expectation is with\n\nrespect to both the internal randomisation of the algorithm and the random loss sequence (cid:96)1:T which\nis drawn from \u00afQ.\n\n(cid:80)K\n\nj=1\n\nK\n\n5\n\n\fThe assumption T (cid:62) K/(4(1 \u2212 \u03b1)) above ensures that \u03b5 (cid:54) 1 \u2212 \u03b1, so that the Qj are well de\ufb01ned.\nProof of Lemma 4. We lower bound the regret by the pseudo-regret for each distribution Qj:\n\nEQj\n\n(cid:35)\n\nT(cid:88)\n\n(cid:34) T(cid:88)\n(cid:96)It,t \u2212 min\nT(cid:88)\n1(cid:54)i(cid:54)K\n\n(cid:34) T(cid:88)\n(cid:2)\u03b1 + \u03b5 \u2212 \u03b51{It=j}(cid:3) \u2212 T \u03b1 = T \u03b5\n\n(cid:62) EQj\n\nEQj\n\n(cid:96)i,t\n\nt=1\n\nt=1\n\nt=1\n\n=\n\n(cid:35)\n(cid:32)\n\n(cid:96)It,t\n\n(cid:34) T(cid:88)\n\nt=1\n\n(cid:35)\n(cid:33)\n\n(cid:96)i,t\n\n\u2212 min\n1(cid:54)i(cid:54)K\n\nEQj\n\nT(cid:88)\n\n1 \u2212 1\nT\n\nQj(It = j)\n\nt=1\n\n(9)\nwhere the \ufb01rst equality follows because EQj [(cid:96)It,t] = EQj [EQj [(cid:96)It,t|(cid:96)1:t\u22121, It]] = EQj [\u03b1 + \u03b5 \u2212\n\u03b51{It=j}] since under Qj, the conditional distribution of (cid:96)t given ((cid:96)1:t\u22121, It) is simply \u2297K\ni=1B(\u03b1 +\n\u03b5 \u2212 \u03b51{i=j}). To bound (9) from below, note that by Pinsker\u2019s inequality we have for all t \u2208\n{1, . . . , T} and j \u2208 {1, . . . , K}, Qj(It = j) (cid:54) Q0(It = j) + (KL(QIt\nj )/2)1/2, where Q0 =\nBer(\u03b1 + \u03b5)\u2297KT is the joint probability distribution that makes all the (cid:96)i,t i.i.d. Ber(\u03b1 + \u03b5), and\nQIt\n0 and QIt\nj denote the laws of It under Q0 and Qj respectively. Plugging the last inequality above\ninto (9), averaging over j = 1, . . . , K and using the concavity of the square root yields\n\n0 , QIt\n\nt=1\n\n,\n\n(cid:34) T(cid:88)\n\nt=1\n\nE \u00afQ\n\n(cid:96)It,t \u2212 min\n1(cid:54)i(cid:54)K\n\nT(cid:88)\n\nt=1\n\n(cid:118)(cid:117)(cid:117)(cid:116) 1\n\n2T\n\n\u2212\n\n\uf8eb\uf8ed1 \u2212 1\n(cid:17)\n\nK\n\n(cid:96)i,t\n\n(cid:62) T \u03b5\n\n(cid:35)\n(cid:80)K\n(cid:16)Q(ht,It)\n(cid:2)Nj(t \u2212 1)(cid:3)\n\nK\n\n0\n\n= EQ0\n\nj\n\n, Q(ht,It)\n\u03b52\n\n\u03b1(1 \u2212 \u03b1)\n\n(cid:17) (cid:54) KL\n\n(cid:54) EQ0\n\nwhere we recall that \u00afQ = 1\nKL(QIt\nFrom Lemma 1\n\n0 , QIt\n\nQj. The rest of the proof is devoted to upper-bounding\nj ). Denote by ht = (Is, (cid:96)Is,s)1(cid:54)s(cid:54)t\u22121 the history available at the beginning of round t.\n\nj=1\n\n(cid:2)Nj(t \u2212 1)(cid:3) KL(cid:0)B(\u03b1 + \u03b5),B(\u03b1)(cid:1)\n\n(cid:16)QIt\n\nKL\n\n0 , QIt\n\nj\n\nT(cid:88)\n\n1\nK\n\nK(cid:88)\n\nKL(cid:0)QIt\n\n0 , QIt\n\nj\n\nt=1\n\nj=1\n\n(cid:1)\uf8f6\uf8f8 ,\n\n(10)\n\nPart 2: Next we prove that\n\nQj(L\u2217\n\nT > T \u03b1) (cid:54) 1\n32T\n\n.\n\n6\n\n,\n\n(11)\n\nwhere the last inequality follows by upper bounding the KL divergence by the \u03c72 divergence (see the\nsupplementary material). Averaging (11) over j \u2208 {1, . . . , K} and t \u2208 {1, . . . , T} and noting that\n\n(cid:80)T\nt=1(t \u2212 1) (cid:54) T 2/2 we get\nK(cid:88)\nPlugging the above inequality into (10) and using the de\ufb01nition of \u03b5 = (1/2)(cid:112)\u03b1(1 \u2212 \u03b1)K/T yields\n\n(t \u2212 1)\u03b52\nK\u03b1(1 \u2212 \u03b1)\n\n2K\u03b1(1 \u2212 \u03b1)\n\n0 , QIt\n\n1\nK\n\nT \u03b52\n\n(cid:54)\n\nj=1\n\n.\n\nj\n\nKL(cid:0)QIt\nT(cid:88)\n\nT(cid:88)\n(cid:18)\n\nt=1\n\n(cid:1) (cid:54) 1\n(cid:35)\n\nT\n\n(cid:62) T \u03b5\n\n(cid:19)\n\n(cid:112)\u03b1(1 \u2212 \u03b1)T K .\n\n1 \u2212 1\nK\n\n\u2212 1\n4\n\n(cid:62) 1\n8\n\nE \u00afQ\n\n(cid:96)It,t \u2212 min\n1(cid:54)i(cid:54)K\n\n(cid:96)i,t\n\nt=1\n\n1\nT\n\nT(cid:88)\n(cid:34) T(cid:88)\n\nt=1\n\nt=1\n\nProof of Theorem 3. We show that there exists a loss sequence (cid:96)1:T \u2208 [0, 1]KT such that L\u2217\n(cid:54) \u03b1T\n\u221a\nand E[RT ((cid:96)1:T )] (cid:62) (1/27)\n\u03b1T K. Lemma 4 above provides such kind of lower bound, but without\nthe guarantee on L\u2217\nT . For this purpose we will use Lemma 4 with a smaller value of \u03b1 (namely, \u03b1/2)\nand combine it with Bernstein\u2019s inequality to prove that L\u2217\nPart 1: Applying Lemma 4 with \u03b1/2 (note that T (cid:62) K (cid:62) K/(4(1 \u2212 \u03b1/2)) by assumption on T )\nand noting that maxj EQj [RT ((cid:96)1:T )] (cid:62) E \u00afQ[RT ((cid:96)1:T )] we get that for some j \u2208 {1, . . . , K} the\n\nprobability distribution Qj de\ufb01ned with \u03b5 = (1/2)(cid:112)(\u03b1/2)(1 \u2212 \u03b1/2)K/T satis\ufb01es\n\n(cid:54) T \u03b1 with high probability.\n\nT\n\nT\n\n(cid:114) \u03b1\n\n(cid:16)\n\n(cid:17)\n\n1 \u2212 \u03b1\n2\n\nT K (cid:62) 1\n32\n\n2\n\n\u221a\n\nEQj [RT ((cid:96)1:T )] (cid:62) 1\n8\n\nsince \u03b1 (cid:54) 1/2 by assumption.\n\n6\u03b1T K\n\n(12)\n\n(13)\n\n\ft=1 (cid:96)j,t. Second, note that under Qj, the (cid:96)j,t, t (cid:62) 1, are i.i.d.\nTo this end, \ufb01rst note that L\u2217\nBer(\u03b1/2). We can thus use Bernstein\u2019s inequality: applying Theorem 2.10 (and a remark on p.38)\nof Boucheron et al. [2013] with Xt = (cid:96)j,t \u2212 \u03b1/2 (cid:54) 1 = b, with v = T (\u03b1/2)(1 \u2212 \u03b1/2), and with\nc = b/3 = 1/3), we get that, for all \u03b4 \u2208 (0, 1), with Qj-probability at least 1 \u2212 \u03b4,\n\nT\n\n(cid:16)\n\n(cid:17)\n\n2T\n\n\u03b1\n2\n\n1 \u2212 \u03b1\n2\n\nlog\n\n1\n\u03b4\n\n+\n\n1\n3\n\nlog\n\n1\n\u03b4\n\n(cid:54)(cid:80)T\n\n(cid:114)\n(cid:19)(cid:114)\n\n+\n\n(cid:96)j,t (cid:54) T \u03b1\n2\n\n(cid:18)\n\nL\u2217\n\nT\n\n(cid:54) T(cid:88)\n\nt=1\n\n(cid:54) T \u03b1\n2\n\n1\n3\n\n1\n\u03b4\n\n(cid:54) T \u03b1\n2\n\nT \u03b1\n2\n\nT\n\n+\n\n1 +\n\nT \u03b1 log\n\n(14)\nwhere the second last inequality is true whenever T \u03b1 (cid:62) log(1/\u03b4) and that last is true whenever\nT \u03b1 (cid:62) (8/3)2 log(1/\u03b4) = c log(1/\u03b4). By assumption on \u03b1, these two conditions are satis\ufb01ed for\n\u03b4 = 1/(32T ), which concludes the proof of (13).\nConclusion: We show by contradiction that there exists a loss sequence (cid:96)1:T \u2208 [0, 1]KT such that\nL\u2217\n\n(cid:54) \u03b1T and\n\n= T \u03b1 ,\n\n+\n\nE[RT ((cid:96)1:T )] (cid:62) 1\n64\n\n6\u03b1T K ,\n\n(15)\nwhere the expectation is with respect to the internal randomisation of the algorithm. Imagine for\na second that (15) were false for every loss sequence (cid:96)1:T \u2208 [0, 1]KT satisfying L\u2217\n(cid:54) \u03b1T . Then\nwe would have 1{L\u2217\n6\u03b1T K almost surely (since the internal\nsource of randomness of the bandit algorithm is independent of (cid:96)1:T ). Therefore by the tower rule for\nthe \ufb01rst expectation on the r.h.s. below, we would get\n+ EQj\n\n\u221a\n(cid:54)\u03b1T}EQj [RT ((cid:96)1:T )|(cid:96)1:T ] (cid:54) (1/64)\n(cid:104)\n(cid:104)\n\nRT ((cid:96)1:T )1{L\u2217\n\n(cid:105)\n\n(cid:105)\n\nT\n\nT\n\n\u221a\n\nEQj [RT ((cid:96)1:T )] = EQj\n\u221a\n(cid:54) 1\n64\n\n(cid:54)\u03b1T}\nRT ((cid:96)1:T )1{L\u2217\n6\u03b1T K + T \u00b7 Qj(L\u2217\n\nT\n\nT > T \u03b1) (cid:54) 1\n64\n\n6\u03b1T K (16)\n\u221a\n6\u03b1T K since \u03b1 (cid:62) K/(2T ) >\nwhere (16) follows from (13) and by noting that 1/32 < (1/64)\n4/(6T ) (cid:62) 4/(6T K). Comparing (16) and (12) we get a contradiction, which proves that there exists\na loss sequence (cid:96)1:T \u2208 [0, 1]KT satisfying both L\u2217\n(cid:54) \u03b1T and (15). We conclude the proof by\n6/64 (cid:62) 1/27. Finally, the condition T (cid:62) K \u2228 118 is suf\ufb01cient to make the interval\nnoting that\n\n(cid:2)(c log(32T ) \u2228 (K/2))/T, 1\n\n(cid:3) non empty.\n\n6\u03b1T K +\n\n\u221a\n\n<\n\nT\n\n2\n\n\u221a\n\nT >\u03b1T}\n1\n32\n\n\u221a\n\n1\n32\n\n4 Second-Order Lower Bounds\n\nWe start by giving a lower bound on the regret in terms of the quadratic variation that is close to\nexisting upper bounds except in the dependence on the number of arms. Afterwards we prove that\nbandit strategies cannot adapt to losses that lie in a small range or the existence of an action that is\nalways optimal.\n\nLower bound in terms of quadratic variation We prove a lower bound of \u2126(\n\u03b1T K) over any\nsmall-variation ball V\u03b1,T (as de\ufb01ned by (7)) for all \u03b1 = \u2126(log(T )/T ). This minimax lower bound\n\nmatches the upper bound of Corollary 2 up to a multiplicative factor of K 2(cid:112)log(T ). Closing this\n\ngap is left as an open question, but we conjecture that the upper bound is loose (see also the COLT\nopen problem by Hazan and Kale [2011a]).\nTheorem 4. Let K (cid:62) 2, T (cid:62) (32K) \u2228 601, and \u03b1 \u2208 [(2c1 log(T ) \u2228 8K)/T, 1/4],\n\u221a\nwhere c1 = (4/9)2(3\nThen for any randomised bandit algorithm,\n\u03b1T K/25, where the expectation is taken with respect to the internal\nsup(cid:96)1:T \u2208V\u03b1,T\nrandomisation of the algorithm.\n\nE[RT ((cid:96)1:T )] (cid:62) \u221a\n\n5 + 1)2 (cid:54) 12.\n\n\u221a\n\nThe proof is very similar to that of Theorem 3; it also follows from Lemma 4 and Bernstein\u2019s\ninequality. It is postponed to the supplementary material.\n\nImpossibility results\nIn the full-information setting (where the entire loss vector is observed after\neach round) Cesa-Bianchi et al. [2007, Theorem 6] designed a carefully tuned exponential weighting\nalgorithm for which the regret depends on the variation of the algorithm and the range of the losses:\n(17)\n\nE[RT ((cid:96)1:T )] (cid:54) 4(cid:112)VT log K + 4ET log K + 6ET ,\n\n\u2200(cid:96)1:T \u2208 RKT ,\n\n7\n\n\fVT =(cid:80)T\n\nwhere the expectation is taken with respect to the internal randomisation of the algorithm\nand ET = max1(cid:54)t(cid:54)T max1(cid:54)i,j(cid:54)K |(cid:96)i,t \u2212 (cid:96)j,t| denotes the effective range of the losses and\nt=1 VarIt\u223cpt((cid:96)It,t) denotes the cumulative variance of the algorithm (in each round t the\nexpert\u2019s action It is drawn at random from the weight vector pt). The bound in (17) is not closed-form\nbecause VT depends on the algorithm, but has several interesting consequences:\n\n1. If for all t the losses (cid:96)i,t lie in an unknown interval [at, at + \u03c1] with a small width \u03c1 > 0, then\n\nVarIt\u223cpt((cid:96)It,t) (cid:54) \u03c12/4, so that VT (cid:54) T \u03c12/4. Hence\n\nE[RT ((cid:96)1:T )] (cid:54) 2\u03c1(cid:112)T log K + 4\u03c1 log K + 6\u03c1 .\n\nTherefore, though the algorithm by Cesa-Bianchi et al. [2007, Section 4.2] does not use the prior\nknowledge of at or \u03c1, it is able to incur a regret that scales linearly in the effective range \u03c1.\n\n2. If all the losses (cid:96)i,t are nonnegative, then by Corollary 3 of [Cesa-Bianchi et al., 2007] the\n\nsecond-order bound (17) implies the \ufb01rst-order bound\n\n(cid:115)\n\n(cid:18)\n\n(cid:19)\n\nMT \u2212 L\u2217\n\nT\nT\n\nE[RT ((cid:96)1:T )] (cid:54) 4\n\nL\u2217\n\nT\n\nlog K + 39MT max{1, log K} ,\n\n(18)\n\nwhere MT = max1(cid:54)t(cid:54)T max1(cid:54)i(cid:54)K (cid:96)i,t .\n\n3. If there exists an arm i\u2217 that is optimal at every round t (i.e., (cid:96)i\u2217,t = mini (cid:96)i,t for all t (cid:62) 1), then\nany translation-invariant algorithm with regret guarantees as in (18) above suffers a bounded\nregret. This is the case for the fully automatic algorithm of Cesa-Bianchi et al. [2007, Theorem 6]\nmentioned above. Then by the translation invariance of the algorithm all losses (cid:96)i,t appearing in\nthe regret bound can be replaced with the translated losses (cid:96)i,t \u2212 (cid:96)i\u2217,t (cid:62) 0, so that a bound of\nthe same form as (18) implies a regret bound of O(log K).\n4. Assume that the loss vectors (cid:96)t are i.i.d. with a unique optimal arm in expectation (i.e., there\nexists i\u2217 such that E[(cid:96)i\u2217,1] < E[(cid:96)i,1] for all i (cid:54)= i\u2217). Then using the Hoeffding-Azuma inequality\nwe can show that the algorithm of Cesa-Bianchi et al. [2007, Section 4.2] has with high\nprobability a bounded cumulative variance VT , and therefore (by (17)) incurs a bounded regret,\nin the same spirit as in de Rooij et al. [2014], Gaillard et al. [2014].\n\n\u221a\n\ndomised bandit algorithm, sup(cid:96)1,...,(cid:96)T \u2208C\u03c1\n\nWe already know that point 2 has a counterpart in the bandit setting. If one is prepared to ignore\nlogarithmic terms, then point 4 also has an analogue in the bandit setting due to the existence\nof logarithmic regret guarantees for stochastic bandits [Lai and Robbins, 1985]. The following\ncorollaries show that in the bandit setting it is not possible to design algorithms to exploit the range\nof the losses or the existence of an arm that is always optimal. We use Theorem 1 as a general tool\nbut the bounds can be improved to\nT K/30 by analysing the expected regret directly (similar to\nLemma 4).\n\nCorollary 5. Let K (cid:62) 2 and T (cid:62) 32(K \u2212 1) log(14). Then, for any randomised bandit algorithm,\nthere is a loss sequence (cid:96)1:T \u2208 [0, 1]KT such that there exists an arm i\u2217 that is optimal at every\n\nCorollary 4. Let K (cid:62) 2, T (cid:62) 32(K \u2212 1) log(14) and \u03c1 (cid:62) 0.22(cid:112)(K \u2212 1)/T . Then for any ran-\nE[RT ((cid:96)1:T )] (cid:62)(cid:112)T (K \u2212 1)/504, where the expectation\nis with respect to the randomness in the algorithm, and C\u03c1 (cid:44)(cid:8)x \u2208 [0, 1]K : maxi,j |xi \u2212 xj| (cid:54) \u03c1(cid:9).\nround t (i.e., (cid:96)i\u2217,t = mini (cid:96)i,t for all t (cid:62) 1), but E[RT ((cid:96)1:T )] (cid:62) (cid:112)T (K \u2212 1)/504, where the\nthere exists an (cid:96)1:T such that P{RT ((cid:96)1:T ) (cid:62)(cid:112)(K \u2212 1)T log(1/(4 \u00b7 0.15)/27} (cid:62) 0.15/2, which\nimplies (since RT ((cid:96)1:T ) (cid:62) 0 here) that E[RT ((cid:96)1:T )] (cid:62)(cid:112)(K \u2212 1)T /504. Finally note that (cid:96)1:T \u2208\nC\u03c1 since \u03c1 (cid:62)(cid:112)(K \u2212 1) log(1/(4\u03b4))/T /(4\n\nProof of Corollaries 4 and 5. Both results follow from Theorem 1 by choosing \u03b4 = 0.15. Therefore\n\nlog 2) and there exists an i such that (cid:96)i,t (cid:54) (cid:96)j,t for all\n\nexpectation is with respect to the randomness in the algorithm.\n\n\u221a\n\nj and t.\n\nAcknowledgments\n\nThe authors would like to thank Aur\u00e9lien Garivier and \u00c9milie Kaufmann for insightful discus-\nsions. This work was partially supported by the CIMI (Centre International de Math\u00e9matiques et\nd\u2019Informatique) Excellence program. The authors acknowledge the support of the French Agence\nNationale de la Recherche (ANR), under grants ANR-13-BS01-0005 (project SPADRO) and ANR-\n13-CORD-0020 (project ALICIA).\n\n8\n\n\fReferences\nC. Allenberg, P. Auer, L. Gy\u00f6r\ufb01, and G. Ottucs\u00e1k. Hannan consistency in on-line learning in case of\nunbounded losses under partial monitoring. In Proceedings of ALT\u20192006, pages 229\u2013243. Springer,\n2006.\n\nJ. Audibert and S. Bubeck. Minimax policies for adversarial and stochastic bandits. In Proceedings\n\nof Conference on Learning Theory (COLT), pages 217\u2013226, 2009.\n\nP. Auer, N. Cesa-Bianchi, Y. Freund, and R. Schapire. Gambling in a rigged casino: The adversarial\nmulti-armed bandit problem. In Foundations of Computer Science, 1995. Proceedings., 36th\nAnnual Symposium on, pages 322\u2013331. IEEE, 1995.\n\nP. Auer, N. Cesa-Bianchi, Y. Freund, and R.E. Schapire. The nonstochastic multi-armed bandit\n\nproblem. SIAM J. Comput., 32(1):48\u201377, 2002.\n\nS. Boucheron, G. Lugosi, and P. Massart. Concentration inequalities: a nonasymptotic theory of\n\nindependence. Oxford University Press, 2013.\n\nS. Bubeck and N. Cesa-Bianchi. Regret analysis of stochastic and nonstochastic multi-armed bandit\n\nproblems. Foundations and Trends in Machine Learning, 5(1):1\u2013122, 2012.\n\nS. Bubeck, V. Perchet, and P. Rigollet. Bounded regret in stochastic multi-armed bandits.\n\nProceedings of The 26th Conference on Learning Theory, pages 122\u2013134, 2013.\n\nIn\n\nN. Cesa-Bianchi, Y. Mansour, and G. Stoltz. Improved second-order bounds for prediction with\n\nexpert advice. Mach. Learn., 66(2/3):321\u2013352, 2007.\n\nS. de Rooij, T. van Erven, P. D. Gr\u00fcnwald, and W. M. Koolen. Follow the leader if you can, hedge if\n\nyou must. J. Mach. Learn. Res., 15(Apr):1281\u20131316, 2014.\n\nP. Gaillard, G. Stoltz, and T. van Erven. A second-order bound with excess losses. In Proceedings of\n\nthe 27th Conference on Learning Theory (COLT\u201914), 2014.\n\nE. Hazan and S. Kale. A simple multi-armed bandit algorithm with optimal variation-bounded regret.\n\nIn Proceedings of the 24th Conference on Learning Theory, pages 817\u2013820, 2011a.\n\nE. Hazan and S. Kale. Better algorithms for benign bandits. J. Mach. Learn. Res., 12(Apr):1287\u20131311,\n\n2011b.\n\nT. L. Lai and H. Robbins. Asymptotically ef\ufb01cient adaptive allocation rules. Adv. in Appl. Math., 6:\n\n4\u201322, 1985.\n\nG. Neu. Explore no more: Improved high-probability regret bounds for non-stochastic bandits. In\n\nAdvances in Neural Information Processing Systems 28 (NIPS 2015), 2015a.\n\nG. Neu. First-order regret bounds for combinatorial semi-bandits. In Proceedings of The 28th\n\nConference on Learning Theory, pages 1360\u20131375, 2015b.\n\nA. Rakhlin and K. Sridharan. Online learning with predictable sequences. In Proceedings of the 26th\n\nConference on Learning Theory, pages 993\u20131019, 2013.\n\nG. Stoltz. Incomplete information and internal regret in prediction of individual sequences. PhD\n\nthesis, Paris-Sud XI University, 2005.\n\nA. Tsybakov. Introduction to nonparametric estimation. Springer Science & Business Media, 2008.\n\n9\n\n\f", "award": [], "sourceid": 658, "authors": [{"given_name": "S\u00e9bastien", "family_name": "Gerchinovitz", "institution": "Universit\u00e9 Toulouse 3 - Paul Sabatier"}, {"given_name": "Tor", "family_name": "Lattimore", "institution": "Indiana University"}]}