{"title": "Neural Network Analysis of Event Related Potentials and Electroencephalogram Predicts Vigilance", "book": "Advances in Neural Information Processing Systems", "page_first": 651, "page_last": 658, "abstract": null, "full_text": "Neural  Network Analysis  of Event  Related \n\nPotentials and  Electroencephalogram Predicts \n\nVigilance \n\nRita Venturini \n\nWilliam W.  Lytton \n\nTerrence J.  Sejnowski \n\nComputational Neurobiology  Laboratory \n\nThe Salk Institute \nLa J oBa,  CA 92037 \n\nAbstract \n\nAutomated  monitoring  of  vigilance  in  attention  intensive  tasks  such  as \nair  traffic  control  or  sonar  operation  is  highly  desirable.  As  the  opera(cid:173)\ntor monitors the  instrument,  the  instrument  would  monitor the operator, \ninsuring against lapses.  We  have taken a first  step  toward this goal by  us(cid:173)\ning feedforward neural networks trained with backpropagation to interpret \nevent  related  potentials  (ERPs)  and electroencephalogram  (EEG)  associ(cid:173)\nated with periods of high and low vigilance.  The accuracy of our system on \nan ERP data set  averaged  over  28  minutes was  96%,  better than the 83% \naccuracy  obtained  using  linear  discriminant  analysis.  Practical  vigilance \nmonitoring will require prediction over shorter time periods.  We were  able \nto  average  the  ERP  over  as  little  as  2  minutes  and  still  get  90%  correct \nprediction of a vigilance measure.  Additionally, we  achieved similarly good \nperformance  using segments of EEG  power  spectrum  as  short  as  56  sec. \n\n1 \n\nINTRODUCTION \n\nMany  tasks  in  society  demand  sustained  attention  to  minimally  varying  stimuli \nover  a long period of time.  Detection of failure in vigilance during such tasks would \nbe  of enormous  value.  Different  physiological  variables  like  electroencephalogram \n651 \n\n\f652 \n\nVenturini, Lytton,  and Sejnowski \n\n(EEG),  electro-oculogram  (EOG),  heart  rate,  and  pulse  correlate  to  some  extent \nwith  the  level  of  attention  (1,  2,  3).  Profound  changes  in  the  appearance  and \nspectrum of the EEG  with sleep  and  drowsiness  are  well  known.  However,  there  is \nno agreement  as  to which EEG bands can best  predict  changes in vigilance.  Recent \nstudies  (4)  seem  to indicate that there is  a  strong correlation between  several  EEG \npower  spectra  frequencies  changes  and  attentional  level  in  subjects  performing  a \nsustained  task.  Another  measure  that  has  been  widely  assessed  in  this  context \ninvolves  the  use  of event-related  potentials  (ERP)(5).  These  are  voltage  changes \nin  the  ongoing  EEG  that  are  time  locked  to  sensory,  motor,  or  cognitive  events. \nThey  are  usually  too  small to  be  recognized  in  the  background  electrical  activity. \nThe  ERP's signal  is  typically extracted  from  the  background  noise  of the  EEG  as \na  consequence  of averaging over  many trials.  The ERP waveform remains constant \nfor  each  repetition of the event,  whereas  the  background EEG  activity has  random \namplitude.  Late cognitive event-related potentials, like  the  P300,  are well  known to \nbe  related to  attentional allocation (6,  7,8).  Unfortunately,  these  ERPs are evoked \nonly  when  the  subject  is  attending  to  a  stimulus.  This  condition  is  not  present \nin  a  monitoring situation  where  monitoring  is  done  precisely  because  the  time  of \nstimulus  occurrence  is  unknown.  Instead,  shorter  latency  responses,  evoked  from \nunobtrusive  task-irrelevant signals,  need  to be  evaluated. \n\nData from  a  sonar  simulation task  was  obtained  from  S.Makeig  at  al  (9).  They \npresented  auditory  targets  only  slightly  louder  than  background  noise  to  13  male \nUnited  States  Navy  personnel.  Other  tones,  which  the  subjects  were  instructed \nto  ignore,  appeared  randomly  every  2-4  seconds  (task  irrelevant  probes).  Back(cid:173)\nground  EEG  and  ERP  were  both  collected  and  analyzed.  The  ERPs  evoked  by \nthe  task  irrelevant  probes  were  classified  into  two  groups  depending  on  whether \nthey appeared before  a correctly identified target  (pre-hit ERPs) or a missed target \n(pre-lapse  ERPs).  Pre-lapse ERPs showed a  relative increase of P2  and N2  compo(cid:173)\nnents  and  a  decrease  of the  N 1 deflection.  N 1,  N2  and  P2  designate  the  sign  and \ntime of peak  of components  in  the  ERP.  Prior  linear  discriminant  analysis  (LDA) \nperformed on  the  averages  of each  session,  showed  83%  correct  classification  using \nERPs obtained from  a  single scalp  site.  Thus,  the  pre-hit  and pre-lapse  ERPs  dif(cid:173)\nfered  enough  to permit  classification  by  averaging  over  a  large enough  sample.  In \naddition,  EEG  power  spectra over  81  frequency  bands  were  computed.  EEG  clas(cid:173)\nsification was  made on  the basis of a  continuous  measure of performance, the error \nrate,  calculated  as  the mean of hits  and lapses  in  a  32  sec  moving window.  Analy(cid:173)\nsis  of the  EEG  power  spectrum  (9)  revealed  that significant  coherence  is  observed \nbetween  various EEG frequencies  and performance. \n\n2  METHOD \n\n2.1  THE  DATA  SET \n\nTwo different groups of input data were used (ERPs and EEG). For the former, a 600 \nmsec  sample of task  irrelevant  probe  ERP was  reduced  to 40  points  after low-pass \nfiltering.  We normalized the data on the basis of the maximum and minimum values \nof the entire set,  maintaining amplitude variability.  A single  ERP was  classified  as \nbeing  pre-hit  or  pre-lapse  based  on  the  subject's  performance  on  the  next  target \ntone.  EEG  power  spectrum,  obtained  every  1.6  seconds,  was  used  as  an  input  to \n\n\fAnalysis of Event Related Potentials and Electroencephalogram  Predicts Vigilance \n\n653 \n\npredict  a  continuous  estimate  of vigilance  (error  rate),  obtained  by  averaging  the \nsubject's  performance  during  a  32  second  window  (normalized  between  -1  and  1). \nThe  five  frequencies  used  (3,  10,  13,  19  and  39  Hz)  had  previously  shown  to  be \nmost  strongly  related  to error  rate  changes  (9).  Each  frequency  was  individually \nnormalized to range between  -1  and  1. \n\n2.2  THE  NETWORK \n\nFeedforward networks  were  trained  with  backpropagation.  We  compared two-layer \nnetwork  to  three-layer  networks,  varying  the  number  of hidden  units  in  different \nsimulations between  2 and 8.  Each  architecture  was  trained ten times on the same \ntask,  resetting the  weights  every  time with  a  different  random seed.  Initial simula(cid:173)\ntions were  performed to select  network parameter values.  We used a learning rate of \n0.3 divided by  the fan-in and weight initialization in a range between \u00b10.3.  For the \nERP data we  used  a jackknife procedure.  For each simulation, a single  pattern was \nexcluded  from  the training set  and considered  to be  the  test  pattern.  Each pattern \nin  turn  was  removed  and  used  as  the  test  pattern  while  the  others  are  used  for \ntraining.  The EEG  data set was not as  limited as  the  ERP one and the simulations \nwere  performed using half of the data as  training and the remaining part as testing \nset .  Therefore,  for  subjects  that  had two  runs  each,  the  training  and  testing  data \ncame from separate sessions. \n\n3  RESULTS \n\n3.1  ERPs \n\nThe  first  simulation  was  done  using  a  two-layer  network  to  assess  the  adequacy \nof the  neural  network  approach  relative  to  the  previous  LDA  results.  The  data \nset  consisted  of the  grand  averages  of pre-hits  and  pre-lapses,  from  a  single  scalp \nsite  (Cz)  of 9  subjects,  three  of them  with  a  double  session,  giving  a  total  of 24 \npatterns.  The jackknife  procedure  was  done  in two different  ways.  First each  ERP \nwas considered individually, as had been done in the LDA study (pattern-jackknife) . \nSecond all the ERPs of a single subject were grouped together and removed together \nto form the test set  (subject-jackknife).  The network  was  trained for  10,000 epochs \nbefore  testing.  Figure  1 shows  the  weights  for  the  24  networks  each  trained with \na  set  of ERPs  obtained  by  removing  a  single  ERP. The  \"waveform\"  of the  weight \nvalues  corresponds  to features  common to the pre-hit  ERPs and to the negative of \nfeatures  common to  the pre-lapse  ERPs .  Classification of patterns  by  the  network \nwas  considerably  more accurate  than the 83%  correct  that had been obtained with \nthe  previous  LDA  analysis.  96%  correct  evaluation  was  seen  in  seven  of the  ten \nnetworks  started  with  different  random  weight  selections.  The  remaining  three \nnetworks  produced  92%  correct  responses  (Fig.  2).  The  same two  patterns  were \nmissed in all cases.  Using hidden units did not improve generalization.  The subject(cid:173)\njackknife results  were  very  similar:  96%  correct  in two  of ten networks  and 92%  in \nthe  remaining  eight  (Fig.  2).  Thus,  there  was  a  somewhat  increased  difficulty \nin  generalizing  across  individuals.  The  ability  of the  network  to generalize  over  a \nshorter  period  of time was  tested  by  progressively  decreasing  the  number of trials \nused  for  testing  using  a  network  trained  on  the grand  average  ERPs .  Subaverages \n\n\f654 \n\nVenturini,  Lytton, and Sejnowski \n\n0.50 \n\n0.0 \n\n-0.50 \n\n~0~------------~3~0~0------------~600 \n\nrns \n\nFigure 1:  Weights from  24  two-layer networks  trained from different  initial weights: \neach  value  correspond  to a  sample point in time in  the input data. \n\n%CORRECf \nCLASSIFICATION \n\n%CORRECf \nCLASSIFICATION \n\n100 \n\n90 \n\n80 \n\n70 \n\n60 \n\n50 \n\no  24 6  \nHIDDEN UNITS \n\n8 \n\n100 \n\n90 \n\n80 \n\n70 \n\n60 \n\n50 \n\no  2 46  \n\nHIDDEN UNITS \n\n8 \n\nFigure  2:  Generalization  performance  in  Pattern  (left)  and  Subject  (right)  Jack(cid:173)\nknifes,  using  two-layer  and  three-layer  networks  with  different  number  of hidden \nunits.  Each  bar represents  a  different  random start of the  network. \n\n\fAnalysis of Event Related Potentials  and Electroencephalogram  Predicts Vigilance \n\n655 \n\n%  CORRECT \nCLASSIFICA TION \n\n100 -\n\n50 -\n\n0-\n\nI \n1 \n\nI \n32 \n\nI \n64 \n\nI \n96 \n\nI \n\n128 \n\nI \n\n160 \n\nTOTAL NUMBER \n\nFigure  3:  Generalization  for  testing  subaverages  made  using  varymg  number  of \nindividual ERPs \n\nwere  formed  using from  1 to  160  individual  ERPs  (Figure  3) .  Performance with  a \nsingle  ERP  is  at  chance.  With  16  ERPs ,  corresponding  to  about  2  minutes,  90% \naccuracy  was  obtained. \n\n3.2  EEG \n\nWe first  report results using a two-layer network to compare with the previous LDA \nanalysis.  Five  power  spectrum frequency  bands from  a  single  scalp  site  (Cz)  were \nused  as  input  data.  The  error  rate  was  averaged  over  32  seconds  at  1.6  second \nintervals.  In  the  first  set  of runs  both  error  rate  and  power  spectra  were  filtered \nusing  a  two  minute time window.  Good  results  could  be  obtained in cases  where  a \nsubject  made errors  more than 40%  of the  time (Fig.  4).  When the subject  made \nfew  errors,  training  was  more  difficult  and  generalization  was  poor.  These  results \nwere  virtually identical  to the LDA ones.  The lack of improvement is  probably due \nto  the  fact  that  the  LDA  performance  was  already  close  to  90%  on  this  data set. \nUse  of three-layer  networks  did  not improve the generalization performance. \n\nThe  use  of a  running  average  includes  information in  the  EEG  after  the  time  at \nwhich  the  network  is  making a  prediction.  Causal prediction  was  attempted using \nmultiple  power  spectra  taken  at  1.6  sec  intervals  over  the  past  56  sec,  to  predict \nthe  upcoming error  rate.  The  results  for  one  subject  are  shown  in  Figure  5.  The \npredicted  error  rate  differs  from  the  target  with  a  root  mean  square  error  of 0.3. \n\n\f656 \n\nVenturini, Lytton,  and Sejnowski \n\n1.0 -\n\n0.5 \n\n-\n\nE \nR \nR \no \nR \n\nR \nA \nT \nE \n\n:1. r . . ! . \n\nj \nt\\\\\" \n~ ! \nli ., \n\n0.0 _  I \n0.0 \n\nI \n5.0 \n\nI \n15.0 \n\nI \n10.0 \nTIME (min.) \n\nI \n20.0 \n\nI \n25.0 \n\nFigure 4:  Generalization results  predicting error  rate from EEG. The dotted line is \nthe network  output, solid line  the  desired  value. \n\n- I  \n\nE \nR \nR \n0 \nR \n\nR \nA \nT \nE \n\n1.0-\n\n0.5-\n\n0.0-\n\n-0.5-\n\n1 \n0.0 \n\n1 \n5.0 \n\n10.0 \n\n15.0 \nTIME (min.) \n\n1 \n\n20.0 \n\n1 \n\n25.0 \n\n1 \n\n1 \n\nFigure 5:  Causal prediction of error  rate from EEG. The dotted line is the network \noutput,  solid line the  desired  value. \n\n\fAnalysis of Event Related  Potentials and Electroencephalogram  Predicts Vigilance \n\n657 \n\n0.4  -\n\n0.3  -\n\n0.2 -\n\n0.1  -\n\n0.0 -\n\n-0.1  -\n\n-0.2 -\n\n3.05 Hz \n\n9.15 Hz \n\n13.4 Hz \n\n19.5 Hz \n\n39.0 Hz \n\nFigure  6:  Weights  from  a  two-layer  causal  prediction  network.  Each  bar,  within \neach  frequency  band,  represents  the  influence  on  the  output  unit  of power  in that \nband  at previous times ranging from  1 sec  (right  bar)  to  56  sec  (left  bar). \n\nFigure  6 shows  the  weights  from  a  two-layer  network  trained  to predict  instanta(cid:173)\nneous  error  rate.  The network  mostly  uses  information from  the  3.05  Hz  and  13.4 \nHz  frequency  bands  in  predicting  the  error  rate  changes.  The  values  of the  3.05 \nHz  weights  have  a  strong  peak  from  the  most  recent  time  steps,  indicating  that \npower  in  this frequency  band  predicts  the  state of vigilance on  a  short  time scale. \nThe  alternating  positive  and  negative  weights  present  in  the  13.4  Hz  set  suggest \nthat rapid  changes  in  power  in  this  band  might be  predictive  of vigilance  (i.e.  the \nderivative of the power signal). \n\n4  DISCUSSION \n\nThese  results  indicate  that  neural  networks  could  be  useful  in  analyzing  electro(cid:173)\nphysiological  measures.  The EEG  results  suggest  that  the  analysis  can  be  applied \nto  detect  fluctuations  of the  attentional  level  of the  subjects  in  real  time.  EEG \nanalysis  could  also  be  a  useful  tool  for  understanding  changes  that  occur  in  the \nelectric  activity of the  brain during  different  states of attention. \n\nIn the ERP analysis, the lack of improvement with the introduction of hidden units \nmight be  due  to the small size  of the data set.  If the  data set  is  too small, adding \nhidden  units  and  connections  may reduce  the  ability  to  find  a  general  solution  to \nthe problem.  The ERP subject-jackknife  results  point out that inter-subject gener(cid:173)\nalization is  possible.  This suggests  the  possibility of preparing  a  pre-programmed \nnetwork that could  be  used  with multiple subjects  rather than training the network \nfor  each  individual.  The subaverages  results  suggest  that  the  detection  is  possible \n\n\f658 \n\nVenturini, Lytton,  and Sejnowski \n\nin  a relatively brief time interval.  ERPs could be an useful  completion to the EEG \nanalysis in  order to obtain an on line  detector  of attentional changes. \n\nFuture research  will  combination of these  two measures  along with  EOG  and heart \nrate.  The  idea is  to  let  the  model  choose  different  network  architectures  and  pa(cid:173)\nrameters,  depending  on the specific  subtask. \n\nACKNOWLEDGEMENTS \n\nWe  would  like to thank  Scott  Makeig  and  Mark Inlow,  Cognitive  Performance  and \nPsychophysiology  Department,  Naval  Health  Research  Center,  San  Diego  for  pro(cid:173)\nviding the data and for invaluable discussions  and Y.  Le  Cun and L.Y.  Bottou from \nNeuristique  who  provided  the  SN2  simulator.  RV  was  supported  by  Ministry  of \nPublic  Instruction,  Italy;  WWL from  a  Physician Scientist  Award,  National Insti(cid:173)\ntute  of Aging;  TJS  is  an  Investigator  with  the  Howard  Hughes  Medical  Institute. \nResearch  was  supported  by  ONR Grant  N00014-91-J-1674. \n\nREFERENCES \n\n1 Belyavin,  A.  and Wright,  N .A.( 1987).  Changes  in electrical  activity of the  brain \nwith  vigilance.  Electroencephalography  and  Clinical  Neuroscience,  66:137-144. \n\n2 Torsvall,  L.  and  Akerstedt,  T.(1988).  Extreme sleepiness:  quantification of OG \nand spectral  EEG  parameters.  Int.  1.  Neuroscience,  38:435-44l. \n\n3  Fruhstorfer,  H.,  Langanke,  P.,  Meinzer,  K.,  Peter,  J.H.,  and  Pfaff,  U.(1977). \nNeurophysiological vigilance indicators and operational analysis of a train vigilance \nmonitoring  device:  a  laboratory  and  field  study.  In  R.R.Mackie(Ed.),  Vigilance \nTheory,  Operational Performance,  and Physiological Correlates, 147-162, New York: \nPlenum  Press. \n\n4  Makeig,  S.  and  Inlow  M.(1991).  Lapses  in  Alertness:  Coherence  of fluctuations \nin performance and  EEG spectrum.  Cognitive Performance  and Psychophysiology \nDepartment,  NHRC,  San Diego.  Technical  Report. \n\n5 Fruhstorfer, H.  and Bergstrom, R.M.(1969).  Human vigilance and auditory evoked \nresponses.  Electroencephalography  and  Clinical Neurophysiology,  27:346-355. \n\n6  Polich,  J.(1989).  Habituation  of  P300  from  auditory  stimuli.  Psychobiology, \n17:19-28. \n\n7 Polich,  J .(1987).  Task difficulty,  probability, and inter-stimulus interval as  deter(cid:173)\nminants of P300 from auditory stimuli.  Electroencephalography  and  clinical Neuro(cid:173)\nphysiology,  68:311-320. \n\n8 Polich, J.( 1990).  P300,  Probability, and Interstimulus Interval.  Psychophysiology, \n27:396-403. \n\n9  Makeig  S.,  Elliot  F.S.,  Inlow  M.  and  Kobus  D.A.(1991)  Predicting  Lapses  in \nVigilance  Using  Brain  Evoked  Responses  to  Irrelevant  Auditory  Probe.  Cognitive \nPerformance and Psychophysiology Department,  NHRC, San Diego.  Technical  Re(cid:173)\nport. \n\n\f", "award": [], "sourceid": 511, "authors": [{"given_name": "Rita", "family_name": "Venturini", "institution": null}, {"given_name": "William", "family_name": "Lytton", "institution": null}, {"given_name": "Terrence", "family_name": "Sejnowski", "institution": null}]}