{"title": "Linear Hinge Loss and Average Margin", "book": "Advances in Neural Information Processing Systems", "page_first": 225, "page_last": 231, "abstract": null, "full_text": "Linear Hinge Loss and Average Margin \n\nClaudio Gentile \n\nManfred K. Warmuth\u00b7 \n\nDSI, Universita' di Milano, \n\nComputer Science Department, \n\nVia Comelico 39, \n20135 Milano. Italy \n\nUniversity of California, \n95064 Santa Cruz, USA \n\ngentile@dsi.unimi.it \n\nmanfred@cse.ucsc.edu \n\nAbstract \n\nWe  describe a unifying method for proving relative loss  bounds for on(cid:173)\nline linear threshold classification algorithms, such as the Perceptron and \nthe Winnow algorithms.  For classification problems the discrete loss  is \nused,  i.e., the total  number of prediction mistakes.  We  introduce a con(cid:173)\ntinuous loss function, called the \"linear hinge loss\", that can be employed \nto derive the updates of the algorithms.  We  first  prove bounds w.r.t.  the \nlinear hinge loss  and  then  convert them  to  the  discrete  loss.  We  intro(cid:173)\nduce a notion of \"average margin\" of a set of examples .  We  show how \nrelative loss  bounds based on the  linear hinge  loss can  be converted to \nrelative loss bounds i.t.o.  the discrete loss using the average margin. \n\n1  Introduction \nConsider the  classical  Perceptron  algorithm.  The  hypothesis  of this  algorithm  at  trial  t \nis  a  linear threshold function  determined  by  a  weight vector Wt  E  Rn.  For an  instance \nXt  ERn the  linear activation  at  = Wt  .  Xt  is  passed  through  a  threshold  function  (7 r \nwhich is  -Ion arguments less than the threshold rand + 1 otherwise.  Thus the prediction \nof the  algorithm is  binary and  -1 , + 1 denote the  two classes.  The Perceptron algorithm \nis aimed at learning a classification problem where the examples have the form  (X t , Yt )  E \nR n  x  {-I , +1} . \nAfter seeing T  examples  (Xt,Yt)1<t<T,  the  algorithm  predicts with  YT+1  =  (7r(WT +1  . \nxT+ d  on the next instance XT+ 1.  Tfthe algorithm's prediction YT+l  agrees with the label \nYT+ 1  on the instance xT +1,  then  its  loss  is  zero.  If the  prediction and  the label  disagree, \nthen the loss is one. We call this loss the discrete loss. \n\nThe convergence of the  Perceptron algonthm is established in the Perceptron convergence \ntheorem.  There is  a second  by  now classical  algorithm for  learning with  linear threshold \nfunctions : the Winnow algorithm of Nick Littlestone [Lit88] . This algorithm also maintains \na weight vector and predicts with the same linear threshold function defined by the current \nweight vector  Wt .  However, the update  of  the weight vector W t  = (Wt.l , ... , Wt ,n ) \n\n\u2022 Supported by  NSF grant CCR-970020 1. \n\n\f226 \n\nC.  Gentile and M.  K.  Warmuth \n\nperformed by the two algorithms is radically different: \n\nPerceptron:  Wt+l  := Wt  - 1]  6t  Xt \nWinnow:  In  Wt+! ,i  := In  Wt,i  -\n\n1]  6t  Xt ,i \n\nThe Perceptron algorithm performs a simple additive update.  The parameter 1]  is a positive \nlearning rate and  6t  equals (fit  - Yt) /2, which lies in  {-1, 0, + 1 }. When 6t  =  0 the  pre(cid:173)\ndiction of the algorithm is correct and no update occurs.  Both the Perceptron algorithm and \nWinnow update conservatively, i.e., they update only when the prediction of the algorithm \nis  wrong.  If fit  = + 1 and Yt  = -1 then the algorithm overshot and 6t  = + 1.  This causes \nthe  Perceptron  to  subtract  1]  Xt  from  the  current  weight Wt.  Similarly  if fit  = -1 and \nYt  = +1 then the algorithm undershot and 6t  = -1. Now the Perceptron adds 1]  Xt  to  the \ncurrent weight Wt.  We  will  later interpret 6t  Xt  as  a gradient of a loss function.  Winnow \nuses the same gradient but the update is done through the componentwise logarithm of the \nweight vector.  One can also rewrite Winnow's update as \n\nWt+l ,i  := Wt ,i exp (-1] 6tXt,i), \n\ni  =  1, ... , n , \n\nso that the gradient appears in  the exponents of factors that multiply the old  weights.  The \nfactors are now used to correct the weights in  the right direction when the algorithm under \nor overshot. \n\nThe algorithms are good for different purposes and, generally speaking, incomparable (see \n[KWA97]  for  a discussion).  In  [KW97]  a framework  was  introduced for deriving  simple \non-line learning updates. This framework has been applied to a variety of different learning \nalgorithms  and  differentiable  loss  functions  [HKW95,  KW98].  The updates  are  always \nderived by approximately solving the following  minimization problem \n\nWt+!  := argminwU(w),  where U(w)  =  d(w , Wt) + 1]loss(Yt, C1r (w \u00b7 Xt )) . \n\n(1) \nHere loss denotes the chosen loss function.  In  our setting this  would be the discrete loss. \nWhat  is  different  now  is  that the  prediction  of the  algorithm Yt  =  C1r (Wt  .  xd  and  the \ndiscrete loss are discontinuous in  the  weight vector Wt.  We  will return  to  this  point later \nafter discussing the other parts of the above minimization problem. The parameter TJ  is the \nlearning rate mentioned above and,  most importantly, d(w , Wt)  is  a divergence measuring \nhow far W  is from Wt.  The divergence function has two purposes.  It motivates the update \nand it becomes the potential function  in  the amortized analysis used  to  prove loss  bounds \nfor the corresponding algorithm. \n\nThe use of an amortized analysis in the context of learning essentially goes back to  [Lit89] \nand the method for deriving updates based on the divergence was  introduced in  [KW97]. \nThe divergence may be seen as a regularization term and may also serve as  a barrier func(cid:173)\ntion in the optimization problem (1) for the purpose of keeping the weights in  a particular \nregion. The additive algorithms, such as gradient descent and the Perceptron algorithm , use \nd( w, wd  =  Ilw - Wt W /2 as the divergence.  This can be used as  a potential function for \nthe proof of the Perceptron convergence theorem.  Multiplicative update algorithms such as \nWinnow and  various exponentiated gradient algorithms use entropy-based divergences as \npotential functions  [HKW95,  KW98].  The function U in  (1) is  minimized by differentiat(cid:173)\ning w.r.t.  w. This works very well when the loss function is convex and differentiable.  For \nexample for linear regression, when the loss function is  the square loss (Wt  .  Xt  - Yt)2/2 , \nthen minimizing U( w ) with the divergence Ilw - Wt 112/2 gives the Widrow-Hoff update: \n\nWt+!  :=  W t  -1](Wt+l  . Xt  - Yt )X t  ~ Wt  -1](Wt  . Xt  - Yt)Xt. \n\nVarious exponentiated gradient algorithms [KW97]  can be derived in  the same way  when \nentropic  divergences are  used  instead.  However,  in  our case  we  cannot differentiate the \ndiscrete loss since it is discontinuous. \n\nWe  asked ourselves which  loss function  motivates the Perceptron and Winnow algorithms \nin  this  framework.  We  will  see that the  loss function that achieves this is continuous and \n\n\fLinear Hinge Loss and Average Margin \n\n227 \n\nits  gradient w.r.t.  Wt  is  Otxt.  where Ot  E {-I, 0, + I}.  We call this loss the (linear) hinge \nloss (HL) and we believe this is  the key  tool for understanding linear threshold algorithms \nsuch  as  the  Perceptron  and  Winnow.  However,  in  the  process  of changing  the  discrete \nloss to the HL  we  also changed our learning problem from  a classification to  a regression \nproblem.  There  are  now  two  versions  of each  algorithm,  a  classification  version  and  a \nregression version. The classification version predicts with a binary label using its linearly \nthresholded prediction.  The loss function  is  the discrete loss.  The regression  version,  on \nthe other hand. predicts on the next instance Xt  with its linear activation at =  Wt \u00b7Xt. In the \nclassification problem the labels Yt  of the examples are -1 and + 1, while in the regression \nproblem the labels at  are  -00 and  +00.  We  will see that both  versions of each algorithm \nuse the  same rule to update the weight vector Wt. \n\nAnother strong hint that the HL is related to  Perceptron and Winnow comes from  the fact \nthat this loss may be seen as a limiting case of the entropic loss used in logistic regression. \nIn  logistic regression the threshold function  Ur  is  replaced by  the smooth  tanh function. \nThere is  a technical way of associating a \"matching loss function\" with a given increasing \ntransfer function  [HKW95].  The matching  loss for the  tanh transfer function  is  the  en(cid:173)\ntropic loss.  We  will  show that  by  making this  transfer function  steeper and by taking  the \nright viewpoint of the matching loss, the entropic loss converges to the HL. In the limiting \ncase the slope ofthe transferfunction is infinite, i.e., it becomes the threshold function U r \u2022 \n\nThe question  is  whether  this  introduction  of the  HL  buys  us  anything.  We  believe  so. \nWe can prove a unifying meta-theorem for the whole class of general additive algorithms \n[GLS97, KW98], when defined w.r.t.  the HL. The bounds for the regression versions of the \nPerceptron and Winnow are simple special cases.  These loss bounds can then be converted \nto  loss bounds for the corresponding classification problems w.r.t.  the discrete loss.  This \nconversion is carried out through working with the \"average margin\" of a set of examples \nrelative to  a  linear threshold classifier.  The conversion of the HL described in  this  paper \ncan then be considered a principled way of deriving average margin-based mistake bounds. \nThe average  margin  reveals  the  inner  structure  of mistake  bound  results  that  have  been \nproven thus  far  for conservative on-line  algorithms.  Previously used  definitions,  such  as \nthe  deviation  [FS98]  and  the  attribute  error  [Lit91],  can  easily  be  related  to  the  average \nmargin or reinterpreted in terms of the HL and the average margin. \n\n2  Preliminaries and the linear hinge loss \nWe  define  two  subsets  of n n:  the  weight domain  Wand the  instance  domain  X.  The \nweights W  maintained by the algorithms always lie in the weight domain and the instances \nx  of the examples always lie in the instance domain.  We require W  be convex. \nA  general  additive  algorithm  and  a divergence are defined  in  terms of a  link function  f. \nSuch a function  is  a vector valued function  from the interior int W  of the  weight domain \nW  onto nn.  with  the property that its Jacobian  is  strictly  positive definite everywhere in \nint  W.  A  link function  f  has  a unique  inverse  f- 1  :  nn  -+  int  W.  We  assume  that f \nis  the gradient of a  (potential) function  Pr  from  int  W  to n,  i.e.,  f(w)  =  \\7Pr (w)  for \nW  E int W.  It is easy to extend the domain of Pr such that it includes the boundary of W. \nFor any  link function  f,  a (Bregman) divergence function  dr  :  W  x  int  W  -+  [0, (0)  is \ndefined as  [Bre67]: \n\ndr(u,w) = Pr(u) - Pr(w) - (u - w)\u00b7 f(w). \n\n(2) \nThus dr( u, w) is the difference between Pr{ u) and its first  order Taylor expansion around \nw. Since f  has a strictly positive definite Jacobian everywhere in int W, the potential Pr is \nstrictly convex over W. Thus dr(u, w)  ~ 0 with equality holding iff u  =  w. \nThe Perceptron algorithm is  motivated by the identity link f (w) = w, with weight domain \nW  = nn. The corresponding divergence is dr(u, w) = Ilu - wW /2. For Winnow the \n\n\f228 \n\nC.  Gentile and M.  K.  Warmuth \n\nii=u(a) \n\nO ra   0 \n\nr \n\nUr(a)  =  -1 \n\nUr(a)  =  +1 \nFigure 1:  HL( a, a)  as a function of a for the two \ncases ur{a)  =  -1, +1. \nweight domain is  W  = [O,oo)n.  The link function  is  the componentwise logarithm.  The \ndivergence related to  this  link function is  the un-normalized relative entropy dr( U, w)  = \n2:~=1 Ui In  ~ + Wi  - Ui. Note that now U  E W, but w  must lie in int W. \nThe following key property immediately follows from the definition of the divergence dr. \n\nFigure 2:  The matching loss \nMLu-l (y, f). \n\nLemma!  [KW98JForanyu  E Wandwl,w2  E intW: \n\nIn this paper we focus on a single neuron using a hard threshold as the transfer function (see \nbeginning of the introduction).  We  will  view  such a neuron in  two ways.  In  the standard \nview the neuron is used for binary classification.  It outputs f)  = U r (a)  trying to predict the \ndesired label y using a threshold r. In the new view the neuron is a regressor. It outputs the \nlinear activation a E 'R, and is  trying to predict a E 'R,. \nFor classification  we use the discrete loss  DL(y,f)  = ~Ifj - yl  E  {0,1}.  For regression \nwe use the linear hinge loss (HL) parameterized by  a threshold r: \n\nFor any a, a E R:  HLr{a, a)  := ~(Ur(a) - CTr(a\u00bb(a - r) =  DL(y, fj)la - rl\u00b7 \n\nNote that the arguments in the two losses DL and HLr are switched. This is intentional and \nwill  be discussed later on. \nIt  can  be  easily  shown  that  HLr{w  . x, a)  is  convex  in  wand that  the  gradient of this \n'VwHLr(w, x, a)  =  ~(ur{a) - ur(a\u00bb  X.  Note  that 6  =  (ur(a)  -\nloss  w.r.t.  w  is \nur (a\u00bb/2  can  only  take  the  three  values  0,  -1, and  +1  mentioned  in  the  introduction. \nStrictly speaking, this  gradient is  not defined  when w  . x  equals the  threshold r.  But we \nwill show in the subsequent sections that even in that case 6 x  has the properties we  need. \nFigure 1 provides a graphical representation of HLr . The threshold function CTr  \"transfers\" \nthe linear activation a = w . x  to a prediction f)  which is a hard classification in {-1, +1}. \n(For the remaining discussion of this section we can assume with no loss of generality that \nthe  threshold  r  is  0.)  Smooth  transfer functions  such  as  the  tanh are  commonly  used \nin  neural  networks,  e.g.,  fj  = tanh(a),  and  relative loss bounds have  been  proven when \nthe comparison class consists of single neurons with any  increasing differentiable transfer \nfunction CT  [HKW95, KW98].  However, for this to work a loss function that \"matches\" the \ntransfer function has to be used.  This loss is defined]  as follows  [HKW95] (see Figure 2): \n\nMLu-l (y, fj)  := f:~ll(~; u(z) - y  dz  =  dU-1 (y, f). \n\nThe matching  loss  for  u(z)  = z  is  the  square  loss  (linear regression)  and  the  matching \nloss  for  u(z)  = tanh(z)  is  the  entropic  loss  (logistic  regression),  which  is  defined  as: \nlIn [HKW95] the notation Lu(Y, y)  is  used for the matching loss MLu-l (y, y).  We use here the \nsubscript a -1 instead of a  to stress a connection between the matching loss and the divergence that \nis discussed at the end of this section. \n\n\fLinear Hinge Loss and Average Margin \n\n229 \n\nMLO'-l (y , y)  = ~(1 - y) In  !=~ + t(1 + y) In  !!~ .  The entropic loss is  finite when y  E \n[-1, +1]  and  y =  tanh(a)  E  (-1 , +1).  These  are  the  ranges  for  y  and  Y needed  for \nlogistic  regression.  We  now  want  to  use  this  type  of loss  for  classification  with  linear \nthreshold functions,  i.e.,  when  y, y E  {-I, + I} and  the  slope s  of the  tanh function  is \nincreased until in the limit it becomes the hard threshold (10 .  Obviously, (1-1 ( -1) = - 00 \nand  (1-1 (+ 1)  = +00  for  any  slope s.  Thus  the  matching  loss  is  infinite for  all  slopes. \nAlso, the known relative loss bounds based on the above notion of matching loss grow with \nthe slope of the transfer function. Thus it seems to  be impossible to  use the matching loss \nwhen the transfer function  is  the hard threshold  170.  However,  we  can still  make sense of \nthe matching loss by viewing the neuron as a regressor. The matching loss is now rewritten \nas another Bregman divergence: \n\nMLcr(a,a)  = lil u(z ) - a(a)  dz  =  PO'(a)  - PO'(a)  - (a  - a)a(a)  =  dcr((i , a), \n\n(3) \n\nwhere PO'  is any function such that P;(a)  =  (1(a) . We now increase the slope of the transfer \nfunction  tanh while keeping a and  a  fixed.  In  the  limiting case (hard thr~old (70)  the \nabove  loss  becomes  twice  the  linear hinge  loss  with  threshold  zero,  i.e.,  MLO'o (a, a)  = \n2 HLo(a, a)  =  ((1o(a)  - (1o(a))(a  - 0). Finally, observe that the two  views of the  neuron \nare related to a duality property [AW98]  of Bregman divergences: \n\n(4) \n\n3  The algorithms \n\nIn  this  paper  we  always  associate  two  general  additive  algorithms  with  a  given \nSuch  algo(cid:173)\nlink  function: \nrithms,  given  in  the  next  table,  correspond  to  the  two  views  of  a  linear  thresh(cid:173)\nold  neuron  discussed  in  the  last  section. \nFor  brevity,  we  will  call  the  two  al(cid:173)\ngorithms  \"the  classification  algorithm\"  and  \"the  regression  algorithm\",  respectively. \n\na  classification  algorithm  and  a  regression  algorithm. \n\nGen. add. classification algorithm: \nFor  t  = 1,2, . . . \nInstance:  Xt  E R n \nPrediction:  Yt  =  ar(wt . Xt) \nLabel:  Yt  E {-I, +1} \nUpdate: \nWt+1 =f- 1 (f(wt) - \u00a5{f)t  - Yt)xt} \nDiscrete loss: \nDL(yt. Yt)  =  tlYt - Ytl \n\nGen. add. regression algorithm: \nFor  t  = 1,2, .. . \nInstance:  Xt  E R n \nPrediction:  at  =  Wt  . Xt \nLabel: 2  at  =  Ytoo \nUpdate: \nwt+l=f- 1 (f(wt) - \u00a5(ar(at) -ur (ad)xt) \nLinear hinge loss: \n\nHLr(at , at) = t(ar(at) -ar(at ))(at - r) \n\nThe classification  algorithm  receives  a label Yt  E  {- I, + 1 },  while  the  regression algo(cid:173)\nrithm receives the infinite label at  with the  sign of Yt.  This assures that Yt  =  a r (ad.  The \nclassification  algorithm predicts with  Yt  =  ar(ad, and  the  regression  algorithm  with  its \nlinear activation at. The loss for the classification algorithm is  the discrete loss DL(Yt, Yt), \nwhile for the regression algorithm we  use HLr( at. at) . The updates of the two algorithms \nare equivalent.  The  update of the regression algorithm is  motivated by  the  minimization \nproblem: \n\nWt+1  :=  argminwU(w)  where U( w) =  dr (w, wd + 'T/  HLr(w . Xt. ad. \nthe \n\nBy  setting \ning  equilibrium  equation \nf- 1  (f (wt ) - \u00a5(ar(Wt+l . Xt) -o\"r(at))xt} . We approximately solve this equation by re(cid:173)\nplacing Wt+l ' Xt  by at  =  Wt ' Xt,  i.e.,  Wt+1  =  f- 1  (f(wt)  - \u00a5 (ar(at}-ar (at)) xt) . \n\nthe  minimum  of  U(w):  Wt -\"-1 \n\nthe  gradient  of  U(w )  w.r.t. \n\nthat  holds  at \n\nw \n\nto  zero  we  get \n\nfollow-\n\n2This  is  a short-hand meaning at  =  ;- 00 if Yt  =  ;-1 and at  =  - 00 if Yt  =  -1 . \n\n\f230 \n\nC.  Gentile and M. K.  Warmuth \n\nBoth  versions  of the  Perceptron  and  Winnow  are  obtained  by  using  the  link  functions \nf(w)  = wand f(w) = (In(wd, ... , In(wn )). respectively. \n\n4  Relative loss bounds \n\nThe following lemma relates the hinge loss of the regression algorithm to the hinge loss  of \nan arbitrary linear predictor u. \nLemma 2  For all U E W, W t  E int W,  Xt  EX, at , r  E Rand 1]  > 0: \n\nHLr(at,at)  - HLr(u , xt,at) + HLr(u, xt,at) \n\n= * (dr(u,wt)  - dr(u,wt+1) + dr(wt,wt+1))  =  !(Yt - Yt) (at  - U\u00b7 xt} \n\n(5) \n\nProof.  We have  dr(u, Wt)  - dr(u, Wt+d + dr(wt , Wt+l)  =  (u  - Wt)  . (J(Wt+d  -\nj(wd)  =  (Wt  - u)  .  \u00a5(O\"r(at)  - o\"r(at)) Xt  =  \u00a5(O\"r(at)  - O\"r(at)) (at  - U . Xt)  = \n1]  (HLr(at, at) - HLr(u . Xt, at) + HLr(u . Xt , ad) . The first equality follows Lemma 1 \nand the second follows from  the update rule of the regression algorithm.  The last equality \nuses HLr(at, at)  as a divergence drTr (at , at)  (see (4)) and again Lemma 1.  0 \nBy  summing the first  equality  of (5)  over all  trials t  we could relate the  total  HLr  of the \nregression  algorithm to the total  HLr  of the regressor u.  However,  our goal  is  to  obtain \nbounds on the  number of mistakes  on the classification  algorithm.  It is  therefore natural \nto  interpret u  too  as  a  linear threshold  classifier.  with  the  same threshold  r  used  by  the \nclassification algorithm. We use the second equality of (5) and sum up over all T  trials: \n\nL,;=I  !(Yt - Yt)  (a - u  . Xt)  = * (dr(u, wd - dr(u, wT+d + L,;=I dr(Wt, wt+d). \n\nNote that the sums in the above equality are unaffected by trials in  which no mistake occurs. \nIn such trials. Yt  =  Yt  and Wt+1  =  Wt . Thus the above is equivalent to the following. where \nM  is the set of trials in  which a mistake occurs: \nL,tEM !(Yt - Yt ) (at  - U\u00b7 Xt)  =  ~ (dr(u, wd - dr (u , wT+d + L,tE.vt dr(wt, wt+d). \n\nSince t(Yt -Yt) =  -Yt when t  E J\\It and dr(u , WT+1)  ::::  0 we get the following theorem: \n\nTheorem 3  Let M  ~ {I, .. . ,T} be the set a/trials in  which the  classification algorithm \nmakes a mistake.  Then/or every u  E  W we have \n\nL,tEM Yt  (u . Xt  - at)  ~ ~ (dr(u, wt} + L,tEM dr(wt, wt+d) . 0 \n\nThroughout the rest of this  section the classification  algorithm is compared to  the  perfor(cid:173)\nmance of a linear threshold classifier u  with threshold r  =  O. We  now apply Theorem 3 to \nthe Perceptron algorithm with WI  =  0, giving a bound i.t.o. the average margin of a linear \nthreshold classifier u  with threshold 0 on a trial sequence M: \n\ni'u ,M  :=  ILl L,tEM Yt U . Xt \u00b7 \n\nI A \n\nSince  Yt  at  ~  0  for  t  E  M .  the  I.h.s.  of  the  inequality  of Theorem  3  is  at  least \nM  \"(U,M'  By  the  update rule.  L.\"tEM  dr  wt, Wt+1  =  L.\"tEM  211xtl12 ~  21.Iv1IX2 ' \nI \n\"'2 \nwhere IIxI12  ~ X 2  for t  E  M .  Since in  Theorem 3 u  is  an  arbitrary vector.  we  replace \nu  by A u  therein, and set A =  .x~ 1)  \u2022  When we solve the resulting inequality for 1.1v11  the \ndependence on 1] cancels out. This gives us the following bound on the number of mistakes: \n\n) , , ! i .   2 \n\n!i. \n\nI'U ,M \n\n\" \n\n( \n\nIMI ~ ( 1 1~1 12X)2 \n\nI'U. )vl \n\n\fLinear Hinge Loss and Average Margin \n\n231 \n\nNote that in the usual mistake bound for the Perceptron algorithm the average 'Yu,/vt  is re(cid:173)\nplaced by mintEM Ytu, Xt. 3 Also, observe that the predictions of the Perceptron algorithm \nwith  r  = 0  and WI  =  0  are  not affected  by  1].  Hence the  previous bound holds for  any \n1]  > O. \nNext,  we  apply Theorem 3  to a  normalized version of Winnow.  This version  of Winnow \nkeeps weights in the probability simplex and  is  obtained by  a slight modification of Win(cid:173)\nnow's  link  function.  We  assume r  =  0  and  choose  X  =  {x  E  nn  :  Ilxlloo  ~ Xoo}. \nUnlike the Perceptron algorithm, a Winnow-like algorithm heavily depends on the learning \nrate, so a careful tuning is needed.  One can show (details omitted due to space limitations) \nthat if 1]  is such that 1] 'YU,M  + 1] X 00  -\nWinnow achieves the bound \n\nX2\u00b0o +1)  > 0 then this normalized version of \n\nIn  ( e 2\n\n'1\n\nIMI < \n\n- ,  \n\n1]'YU,M  +1] \n\ndr(u, WI) \nX \nn \n\n00  -\n\nI  (e 2 '1Xoo +1) , \n\n2 \n\nwhere dr( u, wd is the relative entropy between the two probability vectors U  and Wl. \nConclusions:  In the full  paper we study the case when there is  no consistent threshold U \nmore carefully  and  give  more involved bounds for  the Winnow and  normalized Winnow \nalgorithms as well as  for the p-norm Perceptron algorithm [GLS97]. \n\nReferences \n[AW98]  K.  Azoury  and  M.  K.  Warmuth\",  \"Relative  loss  bounds  and  the  exponential \n\n[Bre67] \n\n[FS98] \n\nfamily of distributions\", \"1998\", Unpublished manuscript. \nL.M.  Bregman.  The relaxation method of finding the common point of convex \nsets  and  its  application  to  the  solution  of problems  in  convex  programming. \nUSSR Computational Mathematics and Physics, 7 :200-217, 1967. \ny.  Freund  and  R.  Schapire.  Large  margin  classification  using  the  perceptron \nalgorithm.  In 11th COLT,  pp. 209-217, ACM,  1998. \n\n[GLS97]  A.  J.  Grove,  N.  Littlestone,  and  D.  Schuurmans.  General convergence results \n\nfor linear discriminant updates.  In  10th COLT,  pp.  171-183. ACM,  1997. \n\n[HKW95]  D.  P.  Helmbold, 1.  Kivinen.  and M.  K.  Warmuth .  Worst-case  loss  bounds for \n\n[KW97] \n\n[KW98] \n\nsigmoided linear neurons.  In NIPS 1995, pp. 309-315. MIT Press,  1995. \nJ. Kivinen and M.  K.  Warmuth.  Additive versus exponentiated gradient updates \nfor linear prediction.  Inform.  and Comput.,  132(1): 1-64. 1997. \n1.  Kivinen and M.  K.  Warmuth.  Relative loss  bounds for  multidimensional re(cid:173)\ngression problems.  In NIPS 10, pp. 287-293 . MIT Press,  1998. \n\n[Lit88] \n\n[KWA97]  J.  Kivinen, M.  K.  Warmuth, and P.  Auer.  The perceptron algorithm vs.  winnow: \nlinear  vs.  logarithmic  mistake  bounds  when  few  input  variables  are  relevant. \nArtijiciallntelligence, 97:325-343,1997. \nN.  Littlestone.  Learning  when  irrelevant  attributes  abound:  A  new  Iinear(cid:173)\nthreshold algorithm.  Machine Learning, 2:285-318,  1988. \nN.  Littlestone.  Mistake  Bounds  and  Logarithmic  Linear-threshold  Learning \nAlgorithms. PhD thesis. Umversity of California Santa Cruz,  1989. \nN.  Littlestone.  Redundant noisy attributes, attribute errors, and linear threshold \nlearning using Winnow.  In 4th COLT,  pp.  147-156, Morgan Kaufmann,  1991. \n\n[Lit91 J \n\n[Lit89] \n\n3The average  margin ~IU.M may  be positive even  though u  is  not consistent. \n\n\f", "award": [], "sourceid": 1610, "authors": [{"given_name": "Claudio", "family_name": "Gentile", "institution": null}, {"given_name": "Manfred K.", "family_name": "Warmuth", "institution": null}]}