{"title": "Effects of Stimulus Type and of Error-Correcting Code Design on BCI Speller Performance", "book": "Advances in Neural Information Processing Systems", "page_first": 665, "page_last": 672, "abstract": "From an information-theoretic perspective, a noisy transmission system such as a visual Brain-Computer Interface (BCI) speller could benefit from the use of error-correcting codes. However, optimizing the code solely according to the maximal minimum-Hamming-distance criterion tends to lead to an overall increase in target frequency of target stimuli, and hence a significantly reduced average target-to-target interval (TTI), leading to difficulties in classifying the individual event-related potentials (ERPs) due to overlap and refractory effects. Clearly any change to the stimulus setup must also respect the possible psychophysiological consequences. Here we report new EEG data from experiments in which we explore stimulus types and codebooks in a within-subject design, finding an interaction between the two factors. Our data demonstrate that the traditional, row-column code has particular spatial properties that lead to better performance than one would expect from its TTIs and Hamming-distances alone, but nonetheless error-correcting codes can improve performance provided the right stimulus type is used.", "full_text": "Effects of Stimulus Type and of Error-Correcting\n\nCode Design on BCI Speller Performance\n\nJeremy Hill1\n\nJason Farquhar2\n\nSuzanne Martens1\n\nFelix Bie\u00dfmann1,3\n\nBernhard Sch\u00a8olkopf1\n\n1Max Planck Institute for Biological Cybernetics\n\n{firstname.lastname}@tuebingen.mpg.de\n2NICI, Radboud University, Nijmegen, The Netherlands\n\nJ.Farquhar@nici.ru.nl\n\n3Dept of Computer Science, TU Berlin, Germany\n\nAbstract\n\nFrom an information-theoretic perspective, a noisy transmission system such as a\nvisual Brain-Computer Interface (BCI) speller could bene\ufb01t from the use of error-\ncorrecting codes. However, optimizing the code solely according to the max-\nimal minimum-Hamming-distance criterion tends to lead to an overall increase\nin target frequency of target stimuli, and hence a signi\ufb01cantly reduced average\ntarget-to-target interval (TTI), leading to dif\ufb01culties in classifying the individual\nevent-related potentials (ERPs) due to overlap and refractory effects. Clearly any\nchange to the stimulus setup must also respect the possible psychophysiologi-\ncal consequences. Here we report new EEG data from experiments in which we\nexplore stimulus types and codebooks in a within-subject design, \ufb01nding an in-\nteraction between the two factors. Our data demonstrate that the traditional, row-\ncolumn code has particular spatial properties that lead to better performance than\none would expect from its TTIs and Hamming-distances alone, but nonetheless\nerror-correcting codes can improve performance provided the right stimulus type\nis used.\n\n1 Introduction\n\nThe Farwell-Donchin speller [4], also known as the \u201cP300 speller,\u201d is a Brain-Computer Interface\nwhich enables users to spell words provided that they can see suf\ufb01ciently well. This BCI determines\nthe intent of the user by recording and classifying his electroencephalogram (EEG) in response to\ncontrolled stimulus presentations. Figure 1 shows a general P300 speller scheme. The stimuli are\nintensi\ufb01cations of a number of letters which are organized in a grid and displayed on a screen. In a\nstandard setup, the rows and columns of the grid \ufb02ash in a random order. The intensi\ufb01cation of the\nrow or column containing the letter that the user wants to communicate is a target in a stimulus se-\nquence and induces a different brain response than the intensi\ufb01cation of the other rows and columns\n(the non-targets). In particular, targets and non-targets are expected to elicit certain event-related\npotential (ERP) components, such as the so-called P300, to different extents. By classifying the\nepochs (i.e. the EEG segments following each stimulus event) into targets and non-targets, the target\nrow and column can be predicted, resulting in the identi\ufb01cation of the letter of interest.\n\nThe classi\ufb01cation process in the speller can be considered a noisy communication channel where\nthe sequence of EEG epochs is a modulated version of a bit string denoting the user\u2019s desired letter.\n\n1\n\n\fFigure 1: Schematic of the visual speller system, illustrating the relationship between the spatial\npattern of \ufb02ashes and one possible codebook for letter transmission (\ufb02ash rows then columns).\n\nThese bit strings or codewords form the rows of a binary codebook C, a matrix in which a 1 at\nposition (i, j) means the letter corresponding to row i \ufb02ashed at time-step j, and a 0 indicates that it\ndid not. The standard row-column code, in which exactly one row or exactly one column \ufb02ashes at\nany one time, will be denoted RC. It is illustrated in \ufb01gure 1.\n\nA classi\ufb01er decodes the transmitted information into an output bit string.\nIn practice, the poor\nsignal-to-noise ratio of the ERPs hampers accurate classi\ufb01cation of the epochs, so the output bit\nstring may differ from the transmitted bit string (decoding error). Also, the transmitted string may\ndiffer from the corresponding row in the codebook due to modulation error, for example if the user\nlost his attention and missed a stimulus event. Coding theory tells us that we can detect and correct\ntransmission and decoding errors by adding redundancy to the transmitted bit string. The Hamming\ndistance d is the number of bit positions that differ between two rows in a codebook. The minimum\nHamming distance dmin of all pairs of codewords is related to the error correcting abilities of the\ncode by e = (dmin \u2212 1)/2, where e is the maximum number of errors that a code can guarantee to\ncorrect [9]. In general, we \ufb01nd the mean Hamming distance within a given codebook to be a rough\npredictor of that codebook\u2019s performance.\n\nIn the standard approach, redundancy is added by repeating the \ufb02ashing of all rows and columns R\ntimes. This leads to d = 4R between two letters not in the same row or column and dmin = 2R\nbetween two letters in the same row or column. The RC code is a poor code in terms of minimum\nHamming distance: to encode 36 different letters in 12 bits, dmin = 4 is possible, and the achievable\ndmin increases supra-linearly with the total code length L (for example, dmin = 10 is possible in\nL = 24 bits, the time taken for R = 2 repeats of the RC code).\nHowever, the codes with a larger dmin are characterized by an increased weight compared to the RC\ncode, i.e. the number of 1\u2019s per bitstring is larger. As target stimulus events occur more frequently\noverall, the expected target-to-target interval (TTI) decreases. One cannot approach codebook op-\ntimization, therefore, without asking what effect this might have on the signals we are trying to\nmeasure and classify, namely the ERPs in response to the stimulus events.\n\nThe speller was originally derived from an \u201coddball\u201d paradigm, in which subjects are presented with\na repetitive sequence of events, some of which are targets requiring a different response from the\n(more frequent) non-targets. The targets are expected to evoke a larger P300 than the non-targets.\nIt was generally accepted that the amplitude of the target P300 decreases when the percentage of\ntargets increases [3, 11]. However, more recently, it was suggested that the observed tendency of\nthe P300 amplitude (as measured by averaging over many targets) to decrease with increased target\nprobability may in fact be attributed to greater prevalence of shorter target-to-target intervals (TTI)\n[6] rather than an overall effect of target frequency per se. In a different type of paradigm using only\ntargets, it was shown that at TTIs smaller than about 1 second, the P300 amplitude is signi\ufb01cantly\ndecreased due to refractory effects [15]. Typical stimulus onset asynchronies (SOAs) in the oddball\nparadigm are in the order of seconds since the P300 component shows up somewhere between 200\nand 800 msec[12]. In spellers, small SOAs of about 100 msec are often used [8, 13] in order to\n\n2\n\n\fachieve high information transfer rates. Consequently, one can expect a signi\ufb01cant ERP overlap\ninto the epoch following a target epoch, and since row \ufb02ashes are often randomly mixed in with\ncolumn \ufb02ashes, different targets may experience very different TTIs. For a 6 \u00d7 6 grid, the TTI\nranges from 1\u00d7SOA to 20\u00d7SOA, so targets may suffer to varying degrees from any refractory and\noverlap effects.\n\nIn order to quantify the detrimental effects of short TTI we examined data from the two subjects in\ndataset IIa+b from the BCI Competition III[2]. Following the classi\ufb01cation procedures described in\nsection 3.3, we estimated classi\ufb01cation performance on the individual epochs of both data sets by 10-\nfold cross-validation within each subject\u2019s data set. Binary (target versus non-target) classi\ufb01cation\nresults were separated according to the time since the previous target (TPT)\u2014for the targets this\ndistance measure is equivalent to the TTI. The left panel of \ufb01g 4 shows the average classi\ufb01cation\nerror as a function of TPT (averaged across both subjects\u2014both subjects show the same qualitative\neffect). Evidently, the target epochs with a TPT< 0.5 sec display a classi\ufb01cation accuracy that\napproximates chance performance. Consequently, the target epochs with TPT< 0.5 sec, constituting\nabout 20% of all target epochs in a RC code, do not appear to be useful for transmission [10].\n\nClearly, there is a potential con\ufb02ict between information-theoretic factors, which favour increasing\nthe minimum Hamming distance and hence the overall proportion of target stimuli, and the detri-\nmental psychophysiological effects of doing so.\n\nIn [7] we explored this trade-off to see whether an optimal compromise could be found. We initially\nbuilt a generative model of the BCI system, using the competition data illustrated in \ufb01gure 4, and\nthen used this model to guide the generation and selection of speller code books. The results were\nnot unequivocally successful: though we were able to show effects of both TTIs and of the Hamming\ndistances in our codebooks, our optimized codebook performed no better than the row-column code\nfor the standard \ufb02ash stimulus. However, our series of experiments involved another kind of stim-\nulus, and the effect of our codebook manipulation was found to interact with the kind of stimulus\nused.\n\nThe purpose of the current paper is two-fold:\n\n1. to present new data which ilustrate the stimulus/codebook interaction more clearly, and\ndemonstrate the advantage to be gained by the correct choice of stimulus together with an\nerror-correcting code.\n\n2. to present evidence for another effect, which we had not previously considered in modelling\nour subjects\u2019 responses, which may explain why row-column codes perform better than\nexpected: speci\ufb01cally, the spatial contiguity of rows and columns.\n\n2 Decoding Framework\n\n2.1 Probabilistic Approach to Classi\ufb01cation and Decoding\n\nWe assume an N-letter alphabet \u0393 and an N-letter by L-bit codebook C. The basic demodulation\nand decoding procedure consists of \ufb01nding the letter \u02c6T among the possible letters t \u2208 \u0393 showing\nthe largest probability Pr (t|X) of being the target letter T , given C and the measured brain signals\nX = [x1, . . . , xL], i.e.,\n\n\u02c6T = argmax\n\nPr (t|X) = argmax\n\nt\u2208\u0393\n\nt\u2208\u0393\n\nPr (X|t) Pr (t)\n\nPr (X)\n\n,\n\n(1)\n\nwhere the second equality follows from Bayes\u2019 rule. A simple approach to decoding is to treat the\nindividual binary epochs, with binary labels c = (Ct1 . . . CtL), as independent. This allows us to\nfactor Pr (X|t) into per-epoch probabilities Pr (xj|c) for epoch indices j = 1 . . . L, to give\n\nPr (t|X) =\n\nPr (t)\nPr (X)\n\nL\n\nY\n\nPr (xj|c) =\n\nPr (t)\nPr (X)\n\nL\n\nY\n\nPr (Ctj|xj) Pr (xj)\n\nPr (Ctj)\n\n= ft(X) ,\n\n(2)\n\nj=1\nwhere the second equality again follows from Bayes\u2019 rule.\n\nj=1\n\nThis form of Bayesian decoding [5] forms the basis for our decoding scheme. We train a probabilistic\ndiscriminative classi\ufb01er, in particular a linear logistic regression (LR) classi\ufb01er [1, pp82-85], to\n\n3\n\n\festimate Pr (Ctj|xj) = pj in (2). As a result, we can obtain estimates of the probability Pr (t|X)\nthat a particular letter t corresponds to the user-selected codeword. Note that for decoding purposes\nthe terms Pr (X) and Pr (xj) can be ignored as they are independent of t. Furthermore, the product\nQj Pr (Ctj) depends only on the positive-class prior of the binary classi\ufb01er, Pr (+). In fact, it is\neasy to show that during decoding this term cancels out the effect of the binary prior, which may\ntherefore be set arbitrarily without affecting the decisions made by our decoder. The simplest thing\nto do is to train classi\ufb01ers with Pr (+) = 0.5, in which case the denominator term is constant for all\nt.\n\n2.1.1 Codebook Optimization\n\nWe used a simple model of subjects\u2019 responses in each epoch in order to estimate the probability\nof making a prediction error with the above decoding method. We used it to compute the codebook\nloss, which is the sum of error probabilities, weighted by the probability of transmission of each\nletter. This loss function was then minimized in order to obtain an optimized codebook.\n\nNote that this approach is not a direct attempt to tackle the tendency for the performance of the\nbinary target-vs-nontarget classi\ufb01er to deteriorate when TTI is short (although this would surely be\na promising alternative strategy). Instead, we take a \u201cnormal\u201d classi\ufb01er, as susceptible to short-TTI\neffects as classi\ufb01ers in any other study, but try to estimate the negative impact of such effects, and\nthen \ufb01nd the best trade-off between avoiding short TTIs on the one hand, and having large Hamming\ndistances on the other hand.\n\nSince our optimization did not result in a decisive gain in performance, we do not wish to emphasize\nthe details of the optimization methods here. However, for further details see the supplementary\nmaterial, or our tech report [7]. For the purposes of the current paper it is the properties of the\nresulting codebooks that are important, rather than the precise criterion according to which they are\nconsidered theoretically optimal. The codebooks themselves are described in section 3.1 and given\nin full in the supplementary material.\n\n3 EEG Experiments\n\nWe implemented a Farwell/Donchin-style speller, using a 6 \u00d7 6 grid of alphanumeric characters,\npresented via an LCD monitor on a desk in a quiet of\ufb01ce. Subjects each performed a single 3-hour\nsession during which their EEG signals were measured using a QuickAmp system (BrainProducts\nGmbH) in combination with an Electro-Cap. The equipment was set up to measure 58 channels of\nEEG, one horizontal EOG at the left eye, one bipolar vertical EOG signal, and a synchronization\nsignal from a light sensor attached to the display, all sampled at 250 Hz. We present results from 6\nhealthy subjects in their 20s and 30s (5 male, 1 female).\n\nTwo factors were compared in a fully within-subject design: codebook and stimulus. These are\ndescribed in the next two subsections.\n\n3.1 Codebook Comparison\n\nIn total, we explored 5 different stimulus codes:\n\n1. RCmix: the 12-bit row-column code, with the 12 bits randomly permuted in time (row events\n\nmixed up randomly between column events) as in the competition data [2].\n\n2. RCsep: the 12-bit row-column code, where the 6 rows are intensi\ufb01ed in random order, and\n\nthen the 6 columns in random order.\n\n3. RC\u2217: this code was generated by taking code RCsep and randomizing the assignment be-\ntween codewords and letters. Thus, the TTI and Hamming-distance content of the code-\nbook remained identical to RCsep, but the spatial contiguity of the stimulus events was\nbroken: that is to say, it was no longer a coherent row or column that \ufb02ashed during any\none epoch, but rather a collection of 6 apparently randomly scattered letters. However, if a\nsubject were to have \u201ctunnel vision\u201d and be unable to see any letters other than the target,\nthis would be exactly equivalent to RCsep. As we shall see, for the purposes of the speller,\nour subjects do not have tunnel vision.\n\n4\n\n\fcode\nRCmix \u00d72\nRCsep \u00d72\nRC\u2217 \u00d72\nD10\nD8opt\n\nL\n24\n24\n24\n24\n24\n\ndmin\n4\n4\n4\n10\n8\n\nE(d) E(TTI) E(#11) Pr (1)\n0.17\n6.9\n0.17\n6.9\n0.17\n6.9\n11.5\n0.38\n0.32\n10.7\n\n0.4\n0.1\n0.1\n3.1\n0.0\n\n5.4\n6.0\n6.0\n2.5\n3.1\n\nL\n0.60\n0.56\n0.56\n0.54\n0.44\n\nTable 1: Summary statistics for the 24-bit versions of the 5 codebooks used. E(#11) means the\naverage number of consecutive target letters per codeword, and Pr (1) the proportion of targets. L\nis our estimated probability of an error, according to the model (see supplementary material or [7]).\n\n4. D10: a 24-bit code with the largest minimum Hamming distance we could achieve\n(dmin = 10). To make it, our heuristic for codeword selection was to pick the codeword\nwith the largest minimum distance between it and all previously selected codewords. A\nlarge number of candidate codebooks were generated this way, and the criteria for scoring\na completed codebook were (\ufb01rst) dmin and (second, to select among a large number of\ndmin = 10 candidates) the lowest number of consecutive targets.\n\n5. D8opt: a 24-bit code optimized according to our model. The heuristic for greedy codeword\nselection was the mean pairwise codebook loss w.r.t. previously selected codebook entries,\nand the \ufb01nal scoring criterion was our overall codebook loss function.\n\n3.2 Stimulus Comparison\n\nTwo stimulus conditions were compared. In both conditions, stimulus events were repeated with\na stimulus onset asynchrony (SOA) of 167 msec, which as close as our hardware could come to\nrecreating the 175-msec SOA of competition III dataset II.\nFlashes: grey letters presented on a black background were \ufb02ashed in a conventional manner, being\nintensi\ufb01ed to white for 33 msec (two video frames). An example is illustrated in the inset of the left\npanel of \ufb01gure 2.\nFlips: each letter was superimposed on a small grey rectangle whose initial orientation was either\nhorizontal or vertical (randomly determined for each letter). Instead of the letter \ufb02ashing, the rect-\nangle \ufb02ipped its orientation instantaneously by 90\u25e6. An example is illustrated in the inset of the\nright panel of \ufb01gure 2. Our previous experiments had led us to conclude that many subjects perform\nsigni\ufb01cantly better with this stimulus, and \ufb01nd it more pleasant, than the \ufb02ash. As we shall see, our\nresults from this stimulus condition support this \ufb01nding, and indicate a potentially useful interaction\nbetween stimulus type and codebook design.\n\n3.3 Experimental Procedure\n\nThe experiment was divided into blocks, each block containing 20 trials with short (2\u20134 second)\nrest pauses between trials. Each trial began with a red box which indicated to the subject which\nletter (randomly chosen on each trial) they should attend to\u2014this cue came on for a second, and was\nremoved 1 second before the start of the stimulus sequence. Subjects were instructed to count the\nstimulus events at the target location, and not to blink, move or swallow during the sequence. The\nsequence consisted of L = 72 stimulus events, their spatio-temporal arrangement being determined\nby one of the \ufb01ve code conditions. The 12-bit RC codes were repeated six times in order to make the\nlength up to L = 72 (re-randomizing the row and column order on each repetition) and the 24-bit\noptimized codes were repeated three times (reassigning the codewords between repetitions to ensure\nmaximal gap between targets at the end of one repetition and the beginning of the next) likewise to\nensure a total code length of 72 bits.\nEach of the 5 code conditions occurred 4 times per block, the order of their occurrence being ran-\ndomized. For a given block, the stimulus condition was held constant, but the stimulus type was\nalternated between blocks. In total, each subject performed 16 blocks. Thus, in each of the 10\nstimulus \u00d7 code conditions, there were a total of 32 letter presentations or 2304 stimulus events.\n\n5\n\n\f3.3.1 Online Veri\ufb01cation\n\nSubjects did not receive feedback at the end of each trial. However, at the end of the experiment,\nwe gave the subject the opportunity to perform free-spelling in order to validate the system\u2019s perfor-\nmance: we asked each subject whether they would prefer to spell with \ufb02ips or \ufb02ashes, and loaded\na classi\ufb01er trained on all data from their preferred stimulus type into the system. Using the 72-bit\ncodebooks, all subjects were able to spell 5-15 letters with online performance ranging from 90 to\n100%. Our data analysis below is restricted to leave-one-letter-out of\ufb02ine performance, excluding\nthe free-spelled letters.\n\n3.4 Data Analysis\n\nThe 60-channel data, sampled at 250 Hz, were band-pass \ufb01ltered between 0.1 and 8 Hz using a\nFIR \ufb01lter. The data were then cut into 600-msec (150-sample) epochs time-locked to the stimulus\nevents, and these were downsampled to 25 Hz. The data were then whitened in 60-dimensional\nsensor space (by applying a symmetric spatial \ufb01ltering matrix equal to the matrix-square-root of the\ndata covariance matrix, computed across all training trials and time-samples). Finally a linear LR\nclassi\ufb01er was applied [1, pp82-85]. The classi\ufb01er\u2019s regularization hyperparameter C was found by\n10-fold cross-validation within the training set..\n\nOf\ufb02ine letter classi\ufb01cation performance was assessed by a leave-one-letter-out procedure: for a\ngiven code condition, each of the 32 letters was considered in turn, and a probabilistic prediction\nwas made of its binary epoch labels using the above procedure trained only on epochs from the other\n31 letters. These probabilities were combined using the decoding scheme described in section 2.1\nand a prediction was made of the transmitted letter. We varied the number of consecutive epochs of\nthe test letter that the decoder was allowed to use, from the minimum (12 or 24) up to the maximum\n72. For each epoch of the left-out letter, we also recorded whether the binary classi\ufb01er correctly\nclassi\ufb01ed the epoch as a target or non-target.\n\n4 Results and Discussion\n\nEstimates of 36-class letter prediction performance are shown in \ufb01gures 2 (averaged across subjects,\nas a function of codeword length) and 3 (for each individual subject, presenting only the results\nfor 24-bit codewords). The performance of the binary classi\ufb01er on individual epochs is shown in\n\ufb01gure 4.\n\nt\nc\ne\nr\nr\no\nc\n \ns\nr\ne\n\nt\nt\n\ne\n\nl\n \n\n%\n\n100\n\n90\n\n80\n\n70\n\n60\n\n50\n\n40\n\nflashes\n\nRC\n*\nRC\n\nmix\n\nRC\n\nsep\n\nD10\nD8\n\nopt\n\n12\n\n24\n\n36\n\n    0\n\n16.67 33.33   50 msec\n72\n\n60\n\n48\n\nt\nc\ne\nr\nr\no\nc\n \ns\nr\ne\n\nt\nt\n\ne\n\nl\n \n\n%\n\n100\n\n90\n\n80\n\n70\n\n60\n\n50\n\n40\n\nflips\n\nRC\n*\nRC\n\nmix\n\nRC\n\nsep\n\nD10\nD8\n\nopt\n\n12\n\n24\n\n36\n\n    0\n\n16.67 33.33   50 msec\n48\n\n60\n\n72\n\nlength of code (epochs)\n\nlength of code (epochs)\n\nFigure 2: Of\ufb02ine (leave-one-letter-out) 36-class prediction performance as a function of codeword\nlength (i.e. the number of consecutive epochs of the left-out letter that were used to make a predic-\ntion). Performance values (and standard-error bar heights) are averaged across the 6 subjects.\n\nOur results indicated the following effects:\n\n1. Using the Donchin \ufb02ash stimulus, the deleterious effects of short TTIs were clear to see:\nD10 performed far worse than the other codes despite its larger Hamming distances. In\nboth stimulus conditions, the averaged plots of \ufb01gure 2 indicate that RCmix may also be\n\n6\n\n\fsubject 2\n\nRC* RCmix RCsep D10\n\nD8opt\n\nt\nc\ne\nr\nr\no\nc\n \ns\nr\ne\n\nt\nt\n\ne\n\nl\n \n\n%\n\n100\n\n90\n\n80\n\n70\n\n60\n\n50\n\n40\n\nsubject 3\n\nRC* RCmix RCsep D10\n\nD8opt\n\ncodebook\n\nsubject 5\n\nflashes\nflips\n\n \n\n100\n\ncodebook\n\nsubject 6\n\nt\nc\ne\nr\nr\no\nc\n \ns\nr\ne\n\nt\nt\n\ne\n\nl\n \n\n%\n\n90\n\n80\n\n70\n\n60\n\n50\n\n40\n\nRC* RCmix RCsep D10\n\nD8opt\n\ncodebook\n\nsubject 1\n\nRC* RCmix RCsep D10\n\nD8opt\n\ncodebook\n\nsubject 4\n\n100\n\n90\n\n80\n\n70\n\n60\n\n50\n\n40\n\n100\n\n90\n\n80\n\n70\n\n60\n\n50\n\nt\nc\ne\nr\nr\no\nc\n \ns\nr\ne\n\nt\nt\n\ne\n\nl\n \n\n%\n\nt\nc\ne\nr\nr\no\nc\n \ns\nr\ne\n\nt\nt\n\ne\n\nl\n \n\n%\n\nt\nc\ne\nr\nr\no\nc\n \ns\nr\ne\n\nt\nt\n\ne\n\nl\n \n\n%\n\nt\nc\ne\nr\nr\no\nc\n \ns\nr\ne\n\nt\nt\n\ne\n\nl\n \n\n%\n\n100\n\n90\n\n80\n\n70\n\n60\n\n50\n\n40\n\n100\n\n90\n\n80\n\n70\n\n60\n\n50\n\n40\n\nRC* RCmix RCsep D10\n\nD8opt\n\ncodebook\n\n40\n\n \n\nRC* RCmix RCsep D10\n\nD8opt\n\ncodebook\n\nFigure 3: Of\ufb02ine (leave-one-letter-out) 36-class prediction performance when decoding codewords\nof length 24, for each of the subjects in each of the code conditions.\n\n)\n\nl\n\nm\ne\nb\no\nr\np\n\ni\n\n \ny\nr\na\nn\nb\n(\n \ny\nl\nt\nc\ne\nr\nr\no\nc\n \n\nd\ne\n\nl\n\ni\nf\ni\ns\ns\na\nc\n \ns\nh\nc\no\np\ne\n%\n\n \n\n100\n\n95\n\n90\n\n85\n\n80\n\n75\n\n70\n\n65\n\n60\n\n55\n\n50\n\n45\n\n1\n\ncompetition III subjs IIa and IIb\n\n)\n\nl\n\nm\ne\nb\no\nr\np\n\ni\n\n \ny\nr\na\nn\nb\n(\n \ny\nl\nt\nc\ne\nr\nr\no\nc\n \n\nd\ne\n\nl\n\ni\nf\ni\ns\ns\na\nc\n \ns\nh\nc\no\np\ne\n%\n\n \n\nnon\u2212targets\ntargets\n\n3\n\n2\nepochs since previous target\n\n4\n\n5\n\n6+\n\navg\n\n100\n\n95\n\n90\n\n85\n\n80\n\n75\n\n70\n\n65\n\n60\n\n55\n\n50\n\n45\n\n1\n\nour 6 subjects, flashes\n\n)\n\nl\n\nm\ne\nb\no\nr\np\n\ni\n\n \ny\nr\na\nn\nb\n(\n \ny\nl\nt\nc\ne\nr\nr\no\nc\n \n\nd\ne\n\nl\n\ni\nf\ni\ns\ns\na\nc\n \ns\nh\nc\no\np\ne\n%\n\n \n\nnon\u2212targets\ntargets\n\n3\n\n2\nepochs since previous target\n\n5\n\n4\n\n6+\n\navg\n\n100\n\n95\n\n90\n\n85\n\n80\n\n75\n\n70\n\n65\n\n60\n\n55\n\n50\n\n45\n\n1\n\nour 6 subjects, flips\n\nnon\u2212targets\ntargets\n\n3\n\n2\nepochs since previous target\n\n5\n\n4\n\n6+\n\navg\n\nFigure 4: Illustration of effect of TPT on epoch classi\ufb01cation performance, (left) in the data from\ncompetition III dataset II; (middle) in our experiments, averaged across all subjects and code condi-\ntions for blocks in which the \ufb02ash stimulus was used; (right) in our experiments, averaged across the\nsame subjects and code conditions, but for blocks in which the \ufb02ip stimulus was used. The rightmost\ncolumn of each plot shows average classi\ufb01cation accuracy across all epochs (remember that short\nTTIs are relatively uncommon overall, and therefore downweighted in the average).\n\nperforming slightly less well than RCsep, which has longer TTIs. However, the latter effect\nis not as large or as consistent across subjects as it was in our preliminary study [7].\n\n2. Using the Donchin \ufb02ash stimulus, our optimized code D8opt performs about as well as\n\ntraditional RC codes, but does not outperform them.\n\n3. Generally, performance using the \ufb02ip stimulus is better than with the \ufb02ash stimulus.\n4. Using the \ufb02ip stimulus, both D8opt and D10 perform better than the RC codes, and they\nperform roughly equally as well as each other. We interpret this interaction between stim-\nulus type and code type as an indication that the \ufb02ip stimulus may generate rather different\npsychophysiological responses from the \ufb02ash (perhaps stronger primary visual evoked-\npotentials, in addition to the P300) of a kind which is less susceptible to short TTI (the\n\n7\n\n\fcurves in the right panel of \ufb01gure 4 being \ufb02atter than those in the middle panel). A com-\nparative analysis of the spatial locations of discriminative sources in the two stimulus con-\nditions is beyond the scope of the current short report.\n\n5. Despite having identical TTIs and Hamming distances, RC\u2217 performs consistently worse\n\nthan RCsep, in both stimulus conditions.\n\nIn summary, we have obtained empirical support for the idea that TTI (\ufb01nding #1), Hamming dis-\ntance (\ufb01nding #4) and stimulus type (\ufb01nding #3) can all be manipulated to improve performance.\nHowever, our initial attempt to \ufb01nd an optimal solution by balancing these effects was not successful\n(\ufb01nding #2). In the \ufb02ash stimulus condition, the row-column codes performed better than expected,\nmatching the performance of our optimized code. In the \ufb02ip stimulus condition, TTI effects were\ngreatly reduced, making either D8opt or D10 suitable despite the short TTIs of the latter.\nIt seems very likely that the unexpectedly high performance of RCsep and RCmix can be at least partly\nexplained by the idea that they have particular spatial properties that enhance their performance\nbeyond what Hamming distances and TTIs alone would predict. This hypothesis is corroborated by\n\ufb01nding #5. Models of such spatial effects should clearly be taken into account in future optimization\napproaches.\n\nOverall, best performance was obtained with the \ufb02ip stimulus, using either of the two error-\ncorrecting codes, D8opt or D10: this consistently outperforms the traditional row-column \ufb02ash design\nand shows that error-correcting code design has an important role to play in BCI speller develop-\nment.\n\nAs a \ufb01nal note, one should remember that a language model can be used to improve performance in\nspeller systems. In this case, the codebook optimization problem becomes more complicated than\nthe simpli\ufb01ed setting we examined, because the prior Pr (t) in (2) is no longer \ufb02at. The nature of\nthe best codes, according to our optimization criterion, might change considerably: for example, a\nsmall subset of codewords, representing the most probable letters, might be chosen to be particularly\nsparse and/or to have a particularly large Hamming distance between them and between the rest of\nthe codebook, while within the rest of the codebook these two criteria might be considered relatively\nunimportant. Ideally, the language model would be adaptive (for example, supplying a predictive\nprior for each letter based on the previous three) which might mean that the codewords should be\nreassigned optimally after each letter. However, such considerations must remain beyond the scope\nof our study until we can either overcome the TTI-independent performance differences between\ncodes (perhaps, as our results suggest, by careful stimulus design), or until we can model the source\nof these differences well enough to account for them in our optimization criterion.\n\nReferences\n\n[1] Bishop CM (1995) Neural Networks for Pattern Recognition. Clarendon Press, Oxford.\n[2] Blankertz B, et al. (2006) IEEE Trans. Neural Systems & Rehab. Eng. 14(2): 153\u2013159\n[3] Donchin E, Coles MGH (1988) Behavioural and Brain Sciences 11: 357\u2013374\n[4] Farwell LA, Donchin E (1988) Electroencephalography and Clinical Neurophysiology 70: 510\u2013523\n[5] Gestel T, et al. (2002) Neural Processing Letters, 15: 45\u201348\n[6] Gonsalvez CL, Polich J (2002) Psychophysiology 39(3): 388\u201396\n[7] Hill NJ, et al (2008) Technical Report #166, Max Planck Institute for Biological Cybernetics.\n[8] Krusienski DJ, et al. (2006) Journal of Neural Engineering 3(4): 299\u2013305\n[9] MacKay D (2005) Information Theory, Inference, and Learning Algorithms. Cambridge Univ. Press\n[10] Martens SMM, Hill NJ, Farquhar J, Sch\u00a8olkopf B. (2007) Impact of Target-to-Target Interval on Classi\ufb01-\n\ncation Performance in the P300 Speller. Applied Neuroscience Conference, Nijmegen, The Netherlands.\n\n[11] Pritchard WS (1981) Psychological Bulletin 89: 506\u2013540\n[12] Rugg MD, Coles MGH (2002) Electrophysiology of mind. Oxford Psychology Series 25\n[13] Serby H, Yom-Tov E, Inbar GF (2005) IEEE Trans. Neural Systems & Rehab. Eng. 13(1):89-98\n[14] Wolpaw JR, et al. (2002) Clinical Neurophysiology 113: 767\u2013791\n[15] Woods DL, Hillyard SA, Courchesne E, Galambos R. (1980) Science, New Series 207(4431): 655\u2013657.\n\n8\n\n\f", "award": [], "sourceid": 359, "authors": [{"given_name": "Jeremy", "family_name": "Hill", "institution": null}, {"given_name": "Jason", "family_name": "Farquhar", "institution": null}, {"given_name": "Suzanna", "family_name": "Martens", "institution": null}, {"given_name": "Felix", "family_name": "Biessmann", "institution": null}, {"given_name": "Bernhard", "family_name": "Sch\u00f6lkopf", "institution": null}]}