{"title": "Exploiting Generative Models in Discriminative Classifiers", "book": "Advances in Neural Information Processing Systems", "page_first": 487, "page_last": 493, "abstract": null, "full_text": "Exploiting generative models \n\n\u2022 In \n\ndiscriminative  classifiers \n\nTommi  S.  Jaakkola* \n\nDavid  Haussler \n\nMIT Artificial  Intelligence Laboratorio \n\nDepartment of Computer Science \n\n545  Technology Square \nCambridge, MA 02139 \n\nUniversity of California \nSanta Cruz,  CA 95064 \n\nAbstract \n\nGenerative probability models such as hidden  ~larkov models pro(cid:173)\nvide  a  principled  way of treating  missing  information  and  dealing \nwith variable  length sequences.  On the other hand , discriminative \nmethods  such  as  support  vector  machines  enable  us  to  construct \nflexible  decision  boundaries  and  often  result  in  classification  per(cid:173)\nformance superior to that of the model based approaches.  An ideal \nclassifier should  combine these two  complementary approaches.  In \nthis  paper,  we  develop  a  natural  way  of  achieving  this  combina(cid:173)\ntion  by deriving kernel functions for  use in discriminative methods \nsuch  as support vector  machines from  generative probability mod(cid:173)\nels.  We  provide a  theoretical justification for  this  combination  as \nwell as demonstrate a substantial improvement in  the classification \nperformance in  the context of D~A and  protein sequence analysis. \n\n1 \n\nIntroduction \n\nSpeech, vision , text and biosequence data can be difficult to deal with in the context \nof simple statistical classification  problems.  Because  the  examples  to  be  classified \nare often  sequences or  arrays of variable size  that may have  been distorted  in  par(cid:173)\nticular  ways,  it  is  common to estimate  a  generative model  for  such data,  and  then \nuse  Bayes  rule  to  obtain  a  classifier  from  this  model.  However.  many  discrimina(cid:173)\ntive  methods,  which  directly  estimate  a  posterior  probability for  a  class  label  (as \nin  Gaussian  process  classifiers  [5])  or  a  discriminant  function  for  the  class  label \n(as  in  support  vector  machines  [6])  have  in  other  areas  proven  to  be  superior  to \n\n* Corresponding author. \n\n\f488 \n\nT.  S.  Jaakkola and D. Haussler \n\ngenerative models  for  classification  problems.  The  problem  is  that there  has  been \nno systematic way to extract features or  metric  relations between examples for  use \nwith  discriminative  methods  in  the  context  of  difficult  data  types  such  as  those \nlisted above.  Here we  propose a general method for  extracting these discriminatory \nfeatures  using  a  generative  model.  V{hile  the  features  we  propose  are  generally \napplicable,  they are most naturally suited to kernel  methods. \n\n2  Kernel methods \n\nHere  we  provide  a  brief introduction  to  kernel  methods;  see,  e.g.,  [6]  [5]  for  more \ndetails.  Suppose now  that we  have a training set of examples Xl and corresponding \nbinary  labels  51  (\u00b11) .  In  kernel  methods.  as  we  define  them.  the  label  for  a  new \nexample X  is  obtained from a weighted sum of the training labels.  The weighting of \neach training label 52  consists of two parts:  1)  the overall importance of the example \nXl  as  summarized  with  a  coefficient  '\\1  and  2)  a  measure of pairwise  \"similarity\" \nbetween between XI  and  X, expressed in  terms of a  kernel function  K(X2' X).  The \npredicted  label S for  the new example  X  is  derived from  the following  rule: \n\ns ~ sign  ( ~ S, '\\,K(X,. X) ) \n\n(1) \n\nWe  note  that  this  class  of kernel  methods  also  includes  probabilistic  classifiers,  in \n\\vhich  case  the  above  rule  refers  to  the  label  with  the  maximum  probability.  The \nfree  parameters in the classification rule  are the coefficients  '\\1  and to some degree \nalso  the  kernel  function  K .  To  pin  down  a  particular  kernel  method.  two  things \nneed  to  be  clarified.  First, we  must  define  a  classification loss. or equivalently,  the \noptimization  problem  to  solve  to  determine  appropriate values  for  the  coefficients \n'\\1'  Slight  variations  in  the  optimization  problem  can take  us  from  support  vector \nmachines to generalized linear models.  The second and the more important  issue is \nthe  choice  of the  kernel  function  - the  main  topic of this  paper.  \\Ve  begin  with  a \nbrief illustration of generalized  linear  models  as  kernel  methods. \n\n2.1  Generalized  linear models \n\nFor concreteness we consider here only logistic regression models. while emphasizing \nthat  the  ideas  are  applicable  to  a  larger  class  of  models l . \nIn  logistic  regression \nmodels , the probability of the label 5  given the example X  and a  parameter vector \ne is  given  by2 \n\nP(5IX. e)  =  (7  (5eT  X) \n\n(2) \nwhere  (7(z)  =  (1  + e- z) - l  is  the  logistic  function.  To  control  the  complexity  of \nthe  model  when  the  number  of training  examples  is  small  we  can  assign  a  prior \ndistribution  p(e)  over  the  parameters.  \\Ve  assume  here  that  the  prior  is  a  zero \nmean Gaussian with a possibly full  covariance matrix L:.  The maximum  a  posteriori \n(l\\IAP)  estimate  for  the  parameters e given  a  training set  of examples  is  found  by \n\n1 Specifically.  it  applies  to  all  generalized  linear  models  whose  transfer  functions  are \n\nlog-concave. \nan adjustable bias  term  is  included  in  the inner  product eT  X. \n\n2Here  we  assume that  the constant  + 1  is  appended  to  every  feature  vector  X  so  that \n\n\fExploiting Generative Models in  Discriminative Classifiers \n\n489 \n\nmaximizing the following  penalized  log-likelihood: \n\nI: log P(S, IX 1 ,  B)  + log P(B) \n\nwhere  the  constant  c  does  not  depend  on  B.  It is  straightforward  to  show,  simply \nby  taking  the  gradient  with  respect  to  the  parameters,  that  the  solution  to  this \n(concave)  maximization problem can be written as3 \n\n(4) \n\nXote  that  the coefficients  A,  appear  as  weights  on  the  training  examples  as  in  the \ndefinition of the kernel  methods.  Indeed.  inserting the above solution back into the \nconditional probability model gives \n\n(5) \n\nBy  identifying  !..:(X/. X)  =  X;'f.X  and  noting  that  the  label  with  the  maximum \nprobability is the aile that has the same sign as the sum in  the argument.  this gives \nthe decision  rule  (1). \n\nThrough  the  above  derivation ,  we  have  written  the  primal  parameters  B in  terms \nof the dual coefficients  A,.J.  Consequently.  the penalized log-likelihood function can \nbe  also  written  entirely  in  terms  of  A, :  the  resulting  likelihood  function  specifies \nhow  the  coefficients  are to  be optimized.  This  optimization  problem  has  a  unique \nsolution  and  can  be  put  into  a  generic  form.  Also , the  form  of the  kernel  function \nthat  establishes  the  connection  between  the  logistic  regression  model  and  a  kernel \nclassifier  is  rather  specific,  i.e ..  has  the  inner  product  form  K(X,. X)  =  X;'f.X. \nHowever.  as long as  the examples here can  be replaced with feature vectors derived \nfrom  the examples. this form  of the kernel function  is  the most general.  \\Ve  discuss \nthis further  in  the next  section. \n\n3  The  kernel  function \n\nFor  a  general kernel  fUIlction  to  be valid.  roughly speaking it  only needs  to be pos(cid:173)\nitive semi-definite  (see e.g.  [7]).  According to the t-Iercer 's  theorem.  any such  valid \nkernel function  admits a  representation as  a simple inner  product  bet\\\\'een suitably \ndefined  feature  vectors.  i.e ..  !\":(X,.Xj)  =  0\\,0.'\\) .  where  the feature  vectors come \nfrom some fixed  mapping X  -> \u00a2.'\\.  For example.  in  the previous section the kernel \nfunction  had the form  X;'f.Xj '  which is  a simple inner product for  the transformed \nfeature vector  \u00a2 .'\\  =  'f. 1- X. \n\nSpecifying  it  simple  inner  product  in  the  feature  space  defines  a  Euclidean  met(cid:173)\nric  space.  Consequently.  the  Euclidean  distances  between  the  feature  vectors  are \nobtained  directly  from  the  kernel  fUllction:  with  the  shorthand  notation  K ,}  = \n\n3This corresponds to a  Legendre  transformation  of the loss  functions  log a( z) . \n.}This is  possible for  all  those e that could arise as solutions to the maximum  penalized \n\nlikelihood  problem:  in  other words.  for  all  relevant  e. \n\n\f490 \n\nT.  S.  Jaakkola and D.  Haussler \n\nK(Xi , Xj)  we  get  II<Px,  - <PxJ W =  K ti  - 2Ktj + K jj .  In  addition  to  defining  the \nmetric structure in the feature space, the kernel defines  a  pseudo metric in the orig(cid:173)\ninal example space through  D(Xi,Xj) =  II<px.  - <pxJII.  Thus  the kernel  embodies \nprior  assumptions  about  the  metric  relations  between  the  original  examples.  No \nsystematic procedure has been  proposed for  finding  kernel functions,  let alone find(cid:173)\ning ones that naturally handle variable length examples etc.  This is  the topic of the \nnext section. \n\n4  Kernels  from  generative  probability models:  the Fisher \n\nkernel \n\nThe  key  idea  here  is  to  derive  the  kernel  function  from  a  generative  probability \nmodel.  We  arrive at the same kernel function  from  two  different  perspectives,  that \nof enhancing  the  discriminative  power  of the  model  and  from  an  attempt  to  find \na  natural comparison between examples induced  by the generative model.  Both of \nthese  ideas  are developed in  more detail  in  the longer version of this  paper[4]. \n\nWe have seen in the previous section that defining the kernel function  automatically \nimplies  assumptions  about  metric  relations  between  the  examples.  We  argue  that \nthese metric relations should be defined directly from a generative probability model \nP(XIO).  To  capture  the  generative  process  in  a  metric  between  examples  we  use \nthe gradient space of the generative model.  The gradient of the log-likelihood  with \nrespect  to  a  parameter describes  how  that parameter contributes to the  process of \ngenerating  a  particular example5 .  This gradient  space  also  naturally  preserves  all \nthe structural assumptions that  the model  encodes about the generation  process. \n\nTo  develop  this  idea  more generally,  consider a  parametric class of models  P(XIO) , \no E  e.  This  class  of probability  models  defines  a  Riemannian  manifold  Ale  with \na  local  metric  given  by the  Fisher  information  matrix6  I,  where  I  = Ex{UxU{}, \nUs  =  \\1 ()  log P(XIB), and the expectation is over P(XIO)  (see e.g.  [1]).  The gradient \nof the log-likelihood , Us , is  called the Fisher score,  and plays a  fundamental  role in \nour  development.  The local  metric  on  lvle  defines  a  distance  between  the  current \nmodel P(XIO) and a nearby model P(XIO+J).  This distance is given by D(O, 0+15)  = \n~JT 16,  which  also  approximates the  KL-divergence  between  the  two  models  for  a \nsufficiently  small 6. \n\n\\l(} log P(XIB)  maps  an  example  X  into  a  feature  vector \nThe  Fisher  score  Us  = \nthat  is  a  point  in  the  gradient  space of the  manifold  Ale.  We  call  this  the  Fisher \nscore  mapping.  This  gradient  Us  can  be  used  to  define  the  direction  of steepest \nascent  in  log P(X 10)  for  the example X  along  the  manifold,  i.e. , the gradient in  the \ndirection  6  that  maximizes  log P( X 10)  while  traversing  the  minimum  distance  in \nthe manifold  as  defined  by D(O, 0 + 6).  This latter gradient is  known as the natural \ngradient  (see e.g.  [1])  and is  obtained from  the ordinary gradient via <Ps  =  I - I Ux. \nWe  will  call  the  mapping  X  ~ <Px  the  natural  mapping  of examples  into  feature \nvectors7 .  The  natural  kernel  of this  mapping  is  the  inner  product  between  these \n\n5For  the  exponential  family  of  distributions,  under  the  natural  parameterization  (), \nthese gradients, less  a  normalization  constant  that depends on (), form  sufficient  statistics \nfor  the example. \n\n6For  simplicity  we  have  suppressed  the  dependence  of  I  and  Ux  on  the  parameter \n\nsetting (),  or equivalently, on  the position  in  the manifold . \n\n7 Again,  we have suppressed  dependence on  the parameter setting ()  here. \n\n\fExploiting Generative Models in  Discriminative Classifiers \n\nfeature vectors relative to the local  Riemannian metric: \n\n491 \n\n(6) \n\nWe  call  this  the  Fisher kernel owing  to the fundamental  role  played  by the Fisher \nscores in  its definition.  The role of the information matrix is  less significant;  indeed, \nin  the context of logistic  regression  models,  the matrix appearing  in  the middle  of \nthe  feature  vectors  relates  to  the  covariance  matrix  of a  Gaussian  prior,  as  show \nabove.  Thus, asymptotically, the information matrix is  immaterial, and the simpler \nkernel  KU(Xi , Xj)  ex  u.Z, Ux)  provides  a  suitable substitute for  the Fisher kernel. \nWe  emphasize that the Fisher kernel defined  above provides only the basic compar(cid:173)\nison  between  the examples,  defining  what  is  meant by  an  \"inner  product\"  between \nthe examples  when  the examples are objects  of various  t.ypes  (e.g.  variable  length \nsequences).  The  way  such  a  kernel  funct.ion  is  used  in  a  discriminative  classifier \nis  not  specified  here.  Using  the  Fisher  kernel  directly  in  a  kernel  classifier,  for  ex(cid:173)\nample,  amounts to  finding  a  linear  separating  hyper-plane  in  the  natural gradient. \n(or  Fisher score)  feature space.  The examples may not.  be linearly separable in  this \nfeature space even though the natural metric st.ructure is given by t.he Fisher kernel. \nIt  may be advantageous to search in  the space of quadratic  (or  higher  order)  deci(cid:173)\nsion  boundaries,  which is  equivalent  to transforming the Fisher kernel according to \nR(Xt ,  Xj)  =  (1  + K(Xt \u2022 x)))m  and  using  the  resulting kernel k  in  the classifier. \n\\Ve are now ready to state a  few  properties of the Fisher kernel function.  So long as \nthe probability model P(XIB) is suitably regular then the Fisher kernel derived from \nit  is  a)  a valid  kernel function  and b) invariant to any invertible (and differentiable) \ntransformation  of  the  model  parameters.  The  rather  informally  stated  theorem \nbelow  motivates the use of this  kernel function  in  a  classification setting. \n\nTheorem 1  Given  any  suitably  regular  probability  model P(XIB)  with  parameters \nB  and  assuming  that  the  classification  label  is  included  as  a  latent  variable,  the \nFisher  kernel  K(X 1 ,  X))  =  V~,  I-I Ux]  derived  from  this  model  and  employed  in \na  kernel  classifier is.  asymptotically.  never inferior  to  the  MAP  decision  rule  from \nthis  model. \n\nThe  proofs  and  other  related  theorems  are  presented  in  the  longer  version  of this \npaper  [4]. \n\nTo  summarize,  we  have  defined  a  generic  procedure for  obtaining  kernel  functions \nfrom  generative  probability models.  Consequently the benefits  of generative mod(cid:173)\nels  are  immediately  available  to  the  discriminative  classifier  employing  this  kernel \nfunction .  We now  turn the experimental demonstration of the effectiveness of such \na  combined classifier. \n\n5  Experimental results \n\nHere  we  consider  two  relevant  examples  from  biosequence  analysis  and  compare \nthe  performance  of the  combined  classifier  to  the  best  generative  models  used  in \nthese  problems.  vVe  start with  a  DNA splice site  classification  problem,  where  the \nobjective  is  to  recognize  true  splice  sites,  i.e. ,  the  boundaries  between  expressed \nregions (exons)  in  a  gene and  the intermediate regions  (introns) .  The dat.a set used \nin our experiments consisted  of 9350  DNA  fragments  from  C.  elegans.  Each of the \n\n\f492 \n\nT  S.  Jaakkola and D.  Haussler \n\n2029  true examples is  a  sequence X  over the  DNA  alphabet  {A, G, T, C}  of length \n25;  the  7321  false  examples  are  similar  sequences  that  occur  near  but  not  at  5' \nsplice sites.  All recognition rates we  report on this data set are averages from 7-fold \ncross-validation. \n\nTo  use  the  combined  classifier  in  this  setting  requires  us  to  choose  a  generative \nmodel  for  the  purpose  of deriving  the  kernel  function.  In  order  to test  how  much \nthe performance of the combined classifier depends on the quality of the underlying \ngenerative  model,  we  chose  the  poorest  model  possible.  This  is  the  model  where \nthe DKA residue in each position in the fragment  is  chosen independently of others, \ni.e.,  P(XIB)  = n;!l P(XzIBz)  and , furthermore , the parameters Bz  are set such that \nP( Xzl OJ)  =  1/4 for  all  i  and  all  Xl  E  {A. G, T, C} .  This  model  assigns  the  same \nprobability  to  all  examples  X.  We  can  still  derive  the  Fisher  kernel  from  such  a \nmodel and use it  in  a  discriminative classifier.  In this case we  used  a  logistic regres(cid:173)\nsion model  as  in  (5)  with a  quadratic  Fisher kernel  K(X/. X j ) =  (1 + K(Xz, Xj))2. \nFigure  1 shows  the  recognition  performance of this  kernel  method,  using  the  poor \ngenerative  model,  in  comparison  to  the  recognition  performance  of a  naive  Bayes \nmodel  or  a  hierarchical  mixture  model.  The  comparison  is  summarized  in  ROC \nstyle  curves  plotting  false  positive  errors  (the  errors  of  accepting  false  examples) \nas  a  function  of false  negative  errors  (the  errors  of  missing  true  examples)  when \nwe  vary the classification  bias for  the  labels.  The curves show  that even  with  such \na  poor  underlying  generative  model,  the  combined  classifier  is  consistently  better \nthan either of the  better generative models  alone. \n\nIn  the second  and  more serious  application of the  combined  classifier.  we  consider \nthe well-known  problem of recognizing  remote  homologies  (evolutionary/structural \nsimilarities) between protein sequences8  that have low residue identity.  Considerable \nrecent  work  has  been  done  in  refining  hidden  l\\Iarkov  models  for  this  purpose  as \nreviewed  in  [2],  and  such  models  current  achieve  the  best  performance.  We  use \nthese state-of-the-art HMMs as comparison cases and also as sources for  deriving the \nkernel function.  Here we  used logistic regression with the simple kernel K u (X1 ' X J)' \nas  the number of parameters in  the Hj\\IMs was  several thousand. \n\nThe experiment was set up as follows.  We picked a  particular superfamily (glycosyl(cid:173)\ntransferases)  from  the TIl'vI-barrel fold  in  the SCOP  protein structure classification \n[3],  and  left out one of the four  major families  in  this superfamily for  testing  while \ntraining  the  HMJlvI  as  well  as  the  combined  classifier  on  sequences  corresponding \nto  the remaining  three families .  The false  training examples for  the discriminative \nmethod came from  those sequences in  the same fold  but not  in  the same superfam(cid:173)\nily.  The test sequences consisted of the left-out family  (true examples)  and proteins \noutside the TIM barrel fold  (false examples).  The number of training examples var(cid:173)\nied  around  100 depending on  the  left-out  family.  As  the sequences among the  four \nglycosyltransferase  families  are  extremely  different,  this  is  a  challenging  discrimi(cid:173)\nnation  problem.  Figure lc shows the recognition  performance curves for  the HMM \nand  the corresponding kernel  method,  averaged over  the four-way cross validation. \nThe combined  classifier  yields  a  substantial  improvement  in  performance over  the \nHJl..IM  alone. \n\n8These  are  variable  length  sequences  thus  rendering  many  discriminative  methods \n\ninapplicable. \n\n\fExploiting Generative Models in Discriminative Classifiers \n\n493 \n\n022 \n\n02 \n\n0'. \n\n!!O16 \n\n;01. \n1012 \n~  0' \n~ \n\n0 08 \n\n006 \n\n004 \n\na)  0020 \n\n002 \n\n0 04 \n\n006 \n\nFaN t'leQllttve  tata \n\n022 \n\n0 2 \n\n.016 \n~O ,. \n'to 12 \ni  0 ' \n\n~008 \n\n006 \n\n004 \n\n008 \n\n0'  b)  0020 \n\n002 \n\n004 \n\n006 \n\nFalse I'Wgalllle rate \n\n002' \n\n002 \n\n~ \n\n0015 \n\n00' \n\n000' \n\n00 \n\n0\" \n\n06 \nla1H~_rala \n\n008 \n\n0' \n\nc) \n\nFigure  1:  a)  & b)  Comparison  of classification  performance  between  a  kernel  clas(cid:173)\nsifiers  from  t he  uniform  model  (solid  line)  and  a  mixture  model  (dashed  line) .  In \na)  t he  mixt ure  model  is  a  naive  Bayes  model  and  in  b)  it  has t hree  components  in \neach  class.  c)  Comparison  of homology recognition  performance  between  a  hidden \nMar kov  model  (dashed  line)  and  t he corresponding  kernel classifier  (solid  line). \n\n6  Discussion \n\nThe model based  kernel function derived in this paper provides a generic mechanism \nfor  incorporating  generative  models  into  discriminative  classifiers.  For  discrimina(cid:173)\ntion,  the resulting combined classifier is guaranteed to be superior t o the generative \nmodel  alone  wit h  little  addit ional  computational cost .  Vie not e  that  t he  power  of \nt he  new  classifier  arises  to  a  large  ext.ent  from  the  use  of  Fisher  scores  as features \nin  place of original exa mples.  It is  possible  to  use t hese features  with any classifier. \ne.g.  a  feed-forward  neural  net,  but  kernel  methods  are  most  naturally  suited  for \nincorporating  them . \n\nF inally  we  note that while  we  have  used  classification  t o  guide  the  development  of \nthe  kernel function,  t he  results  are  directly  applicable  t o  regression. clustering. or \neven  interpolation  problems, all  of which  can  easily  exploit  metric  relations  among \nthe examples  defined  by the  Fisher  kernel. \n\nReferences \n\n[1]  S.-I.  Amari .  Natural  gradient  works  efficient ly  in  learning.  Neural  Computation, \n\n10:251- 276,  1998. \n\n[2]  R.  Durbin , S.  Eddy, A.  K rogh,  and  G . :\\Iitchison .  Biological  Sequence  Analysis:  Prob(cid:173)\n\nabilistic  Models  of  Proteins  and Nucleic  Aczds.  Cambridge  C niversity  Press,  1998. \n\n[3]  T.  Hubbard , A. Murzin , S. Brenner , and  C.  Chothia.  seo?:  a  structural classification \n\nof  proteins  database.  NA R , 25(1) :236- 9,  Jan .  1997. \n\n[4]  T.  S.  Jaakkola  and  D .  Haussler . \n\nt ive  classifiers. \nhttp : //r,l'.lY . ai . mit . edu/ rv tommi. \n\n1998. \n\nExploiting  generative  models  in  discrimina-\nRevised  and  extended  version .  \\Vill  be  available  from \n\n[5]  D .  J.  C.  MacKay. \n\nIntroduction  to  gaussian  processes. \n\n1997.  Available  from \n\nhttp : //wol.ra . phy . cam . ac.uk/mackay/. \n\n[6]  V.  Vapnik.  The  nature  of  statistical  learning  theory.  Springer-Verlag.  1995. \n\n[7]  G.  Wahba .  Spline  models  f or  observational  data.  CB:\\IS-NSF  Regional  Conference \n\nSeries  in  Applied  t>.lathematics ,  1990. \n\n\f", "award": [], "sourceid": 1520, "authors": [{"given_name": "Tommi", "family_name": "Jaakkola", "institution": null}, {"given_name": "David", "family_name": "Haussler", "institution": null}]}