{"title": "Kernel Logistic Regression and the Import Vector Machine", "book": "Advances in Neural Information Processing Systems", "page_first": 1081, "page_last": 1088, "abstract": null, "full_text": "Kernel Logistic Regression and the Import\n\nVector Machine\n\nJi Zhu\n\nDepartment of Statistics\n\nStanford University\nStanford, CA 94305\n\nTrevor Hastie\n\nDepartment of Statistics\n\nStanford University\nStanford, CA 94305\n\njzhu@stat.stanford.edu\n\nhastie@stat.stanford.edu\n\nAbstract\n\nThe support vector machine (SVM) is known for its good performance in\nbinary classi\ufb01cation, but its extension to multi-class classi\ufb01cation is still\nan on-going research issue. In this paper, we propose a new approach\nfor classi\ufb01cation, called the import vector machine (IVM), which is built\non kernel logistic regression (KLR). We show that the IVM not only per-\nforms as well as the SVM in binary classi\ufb01cation, but also can naturally\nbe generalized to the multi-class case. Furthermore, the IVM provides an\nestimate of the underlying probability. Similar to the \u201csupport points\u201d of\nthe SVM, the IVM model uses only a fraction of the training data to index\nkernel basis functions, typically a much smaller fraction than the SVM.\nThis gives the IVM a computational advantage over the SVM, especially\nwhen the size of the training data set is large.\n\nis qualitative and assumes values in a \ufb01nite set \u001a\n\n\u0002\u0001\u0015\u0014\u0016\u0005\b\u0007\u000f\u0014\u0017\n , where the output \u0007\u0019\u0018\nfrom \u001a\n\n. We\n,\nto it. Usually it is assumed that the training data are an\nindependently and identically distributed sample from an unknown probability distribution\n\n1 Introduction\nIn standard classi\ufb01cation problems, we are given a set of training data \u0002\u0001\u0004\u0003\u0006\u0005\b\u0007\t\u0003\u000b\n , \f\u0001\u000e\r\u000f\u0005\u0010\u0007\u000f\r\u0011\n ,\n\u0012\u0013\u0012\u0011\u0012\nwish to \ufb01nd a class\ufb01cation rule from the training data, so that when given a new input \u0001\nwe can assign a class \u001b\n\f\u001d\u001e\u0005\u0010\u001f \n .\nThe support vector machine (SVM) works well in binary classi\ufb01cation, i.e. \u0007\"!$#&%'\u0005\u0013(\u000f) , but\n\f\u00019\n\nweakness of the SVM is that it only estimates *\u0011+-,/.10\n\u0002\u001f;:<(/=\n\f\u00019\n\u0017:\nis often of interest itself, where 2\nof a point being in class ( given \u001d@:A\u0001\n\nis the conditional probability\n. In this paper, we propose a new approach, called\nthe import vector machine (IVM), to address the classi\ufb01cation problem. We show that the\nIVM not only performs as well as the SVM in binary classi\ufb01cation, but also can naturally\nbe generalized to the multi-class case. Furthermore, the IVM provides an estimate of the\n\n\u0002\u0001\u000e\n\u000e3$(54\u0006687 , while the probability2\n\u001d>:?\u0001\u000e\n\nits appropriate extension to the multi-class case is still an on-going research issue. Another\n\nfraction of the training data to index the kernel basis functions. We call these training data\n\n\u0002\u0001\u000e\n . Similar to the \u201csupport points\u201d of the SVM, the IVM model uses only a\n\u0002C$D&\n , while the computational cost\nis the number of import points. Since E does not tend to\n\nprobability2\nimport points. The computational cost of the SVM is B\nof the IVM is B\n\n , where E\n\n&E8\n\n\u0002C\n\n\u001c\n2\n\u001c\n\fincrease as C\n\nincreases, the IVM can be faster than the SVM, especially for large training\ndata sets. Empirical results show that the number of import points is usually much less than\nthe number of support points.\n\nIn section (2), we brie\ufb02y review some results of the SVM for binary classi\ufb01cation and\ncompare it with kernel logistic regression (KLR). In section (3), we propose our IVM\nalgorithm. In section (4), we show some simulation results. In section (5), we generalize\nthe IVM to the multi-class case.\n\n2 Support vector machines and kernel logistic regression\n\nThe standard SVM produces a non-linear classi\ufb01cation boundary in the original input space\nby constructing a linear boundary in a transformed version of the original input space.\nThe dimension of the transformed space can be very large, even in\ufb01nite in some cases.\nThis seemingly prohibitive computation is achieved through a positive de\ufb01nite reproducing\n\n, which gives the inner product in the transformed space.\n\nMany people have noted the relationship between the SVM and regularized function es-\ntimation in the reproducing kernel Hilbert spaces (RKHS). An overview can be found in\nEvgeniou et al. (1999), Hastie et al. (2001) and Wahba (1998). Fitting an SVM is equiva-\nlent to minimizing:\n\nkernel \n\n. The\n\n\u0018\u0005\u0004\n\n\u0005\u0016\u0014\n\n!\u0018\u0017\u001a\u0019\n\n:\u0012\u0011\u0013\b\u0015\u0014\n\n(1)\n\u0018\u0003\u0002\n!\u0018\u001b\nwith \u0004\nclassi\ufb01cation rule is given by *\u0011+-,/.10\nBy the representer theorem (Kimeldorf et al (1971)), the optimal \u0004\n\u0002\u0001\u000e\n\n(2)\n\n\f\u0001\n\n\b\n\u0007\u0006\t\b\u000b\n\r\f\nis the RKHS generated by the kernel \n\f\u00019\n has the form:\n\n3$\u0007\n\u0017\u001a\u0019\n7 .\n\n:\u0015\u0011\u001c\b\n\n\f\u0001\u0004\u0005\u0010\u0001\n\n\u0005\u0016\u0011\n\n\u000e\u0010\u000f\n\n.\n\n\u0003\u001e\u001d\nIt often happens that a sizeable fraction of the C\n\n\u0018\u0003\u0002\n\nvalues of\n\nconsequence of the truncation property of the \ufb01rst part of criterion (1). This seems to be an\nattractive property, because only the points on the wrong side of the classi\ufb01cation boundary,\nand those on the right side but near the boundary have an in\ufb02uence in determining the\nposition of the boundary, and hence have non-zero\nsupport points.\n\n\u0018 \u2019s. The corresponding \u00019\u0018 \u2019s are called\n\nNotice that (1) has the form \u001f! \u0006*&*\n(*\b\n( 3\n\nis plotted in Figure\n1, along with several traditional loss functions. As we can see, the negative log-likelihood\n(NLL) of the binomial distribution has a similar shape to that of the SVM. If we replace\n\n2#\"&.\n\u001f%$\n\n , the NLL of the binomial distribution, the problem\n\",+.-0/\n\nbecomes a KLR problem. We expect that the \ufb01tted function performs similarly to the SVM\nfor binary class\ufb01cation.\n\nin (1) with (\u0003)\n\n'\u0006\n\n. The loss function \n\n3\u0017\u0007\n\n&\u0006\n\n\u0018 can be zero. This is a\n\nThere are two immediate advantages of making such a replacement: (a) Besides giving\n\n(*\b\n\na classi\ufb01cation rule, the KLR also offers a natural estimate of the probability 2\n\"1/\nrally be generalized to the multi-class case through kernel multi-logit regression, whereas\nthis is not the case for the SVM. However, because the KLR compromises the hinge loss\nfunction of the SVM, it no longer has the \u201csupport points\u201d property; in other words, all the\n\n , while the SVM only estimates *\u0011+-,\t.10\n\n(54\u0006687 ; (b) The KLR can natu-\n\n\f\u0001\u000e\n\n\"1/\n\n\f\u00019\n\n\u0018 \u2019s in (2) are non-zero.\n\nKLR is a well studied problem; see Wahba et al. (1995) and references there; see also\nGreen et al. (1985) and Hastie et al. (1990).\n\n(\nC\n\u0014\n\u0001\n\u0003\n\n(\n\u0018\n\u0004\n\f\n\n\u0012\n\u0004\n\u0004\n\u0014\n\u0001\n\u0018\n\n\u0018\n\n\u0012\n\u001d\n\u001d\n\b\n\u001d\n\u0007\n(\n\u0004\n\n\u0007\n\u0004\n\n:\n4\n\n2\n3\n\u001d\n\fBinomial NLL\nSquared Error\nSupport Vector\n\n0\n\n.\n\n3\n\n5\n\n.\n\n2\n\n0\n\n.\n\n2\n\ns\ns\no\nL\n\n5\n\n.\n\n1\n\n0\n\n.\n\n1\n\n5\n\n.\n\n0\n\n0\n\n.\n\n0\n\n-3\n\n-2\n\n-1\n\n0\n\nyf(x)\n\n1\n\n2\n\n3\n\nFigure 1: Several loss functions, \u0002\u0001\u0004\u0003\u0006\u0005\b\u0007\n\t\u000b\u0007\r\f\n\nalgorithm will \ufb01nd a sub-model to approximate the full model (2) given by the KLR. The\nsub-model has the form:\n(3)\n\n\u0002C\n\nD5\n ; to save the computational cost, the IVM\n\nThe computational cost of the KLR is B\n\u0011\u001c\b\nis a subset of the training data #\u0011\u0001\n\n\u0002\u0001\u000e\n1:\n\nwhere \u0014\n\n\u0002\u0001\u0004\u0005\b\u0001\n\u0001\u000e\u0014\u0016) , and the data in \u0014 are called import\n\n\u000e\n\u000f\u0011\u0010\u0013\u0012\n\u00038\u0005\b\u0001\u0015\r\u0019\u0005\n\n\u0012\u0011\u0012\u0013\u0012\n\npoints. The advantage of this sub-model is that the computational cost is reduced, espe-\ncially for large training data sets, while not jeopardizing the performance in classi\ufb01cation.\n\n. Lin et al.\n(1998) divide the training data into several clusters, then randomly select a representative\n. Smola et al. (2000) develope a greedy technique to se-\n, such that the span of\n\nSeveral other researchers have investigated techniques in selecting the subset \u0014\nquentially select E columns of the kernel matrix 0\nfrom each cluster to make up \u0014\n\u0005\u0010\u0001\u0016\u0015&\n\n\u0014\u0018\u0017\nthese E columns approximates the span of 0\n\u0014 well in the Frobenius norm.\n\u0014\u0018\u0017\n\f\u0001\n\u0018\b\u0005\u0010\u0001\n(2001) propose randomly selecting E points of the training data, then\n\f\u0001\n, and expanding the results back up to C\nods uses the output \u0007\u0019\u0018\n\u0018 ). The\nin selecting the subset \u0014\nIVM algorithm uses both the output \u0007/\u0018 and the input \u0001\u000e\u0018\n\nWilliams et al.\nusing the Nystrom method to approximate the eigen-decomposition of the kernel matrix\ndimensions. None of these meth-\n\n(i.e., the procedure only involves \u0001\n\nway that the resulting \ufb01t approximates the full model well.\n\nto select the subset \u0014\n\n\u0005\u0010\u0001\u0016\u0015&\n\n\u0014\u0018\u0017\n\n, in such a\n\n\f\u0001\n\n3 Import vector machine\nFollowing the tradition of logistic regression, we let \u0007/\u0018\n\n#5%\n\n\u0005\u0011(\u0006)\n\nFor notational simplicity, the constant term in the \ufb01tted function is ignored.\n\nfor the rest of this paper.\n\nIn the KLR, we want to minimize:\n\n\u0007\u000f\u0018\n\n\f\u0001\u0015\u0018\n\n\b\u001b\u001a\u001d\u001c\u0016\u001e\u0004\n\n\u0002\u0001\u0015\u0018\n\n\b\n\u0010\n\nFrom (2), it can be shown that this is equivalent to the \ufb01nite dimensional form:\n\n(4)\n\n3\u0004\u001f\n\u0007! \n\n#\"\n\n($ \n\n\b(\n\n(\u0003)\n\n\b%\u001a\r\u001c!\u001e\n\n\b\n\n#\"\n\n'&\n\n\u0004\n\u0001\n\u001d\n\u0018\n\n\u0018\n\n\n\u0018\n7\n\u0014\n\n\u0015\n\n7\n0\n\n\u0018\n7\n\u0014\n!\n\u0019\n:\n3\n\u0014\n\u0001\n\u0018\n\u0002\n\u0003\n0\n\u0004\n\n3\n(\n)\n\n(\n\u0004\n7\n\b\n\n6\n\f\n\u0004\n\f\n\n\u000e\n\u000f\n\u0019\n:\n\n\u001f\n\u001d\n\n\b\n\u001f\n\n\u001f\n\u001d\n\b\n\n6\n\u001f\n\u001d\n \n\u001f\n\u001d\n\f\f\u0001\u000e\u0018\b\u0005\u0010\u0001\n\n\u0014\u0018\u0017\n\n\u0012\u0013\u0012\u0011\u0012\n#\" .\n\n; and the regulariza-\n\n; the regressor matrix '\"\n\n, we set the derivative of \u0019 with respect to \u001f\n\n\u00038\u0005\nwhere \u001f\ntion matrix '&\nTo \ufb01nd \u001f\nequal to 0, and use the Newton-\nRaphson method to iteratively solve the score equation. It can be shown that the Newton-\nRaphson step is a weighted least squares step:\n\u0002\u0001\u0004\u0003\n\b\u000b\n\n(5)\n#\"\n\"\u0007\u0005\n\u0002\u0001\u0004\u0003\nin the \t th step, \u001f\nwhere \u001f\nmatrix is \u0005\nAs mentioned in section 2, we want to \ufb01nd a subset \u0014 of #&\u0001\nfor every subset \u0014\n( ) Let \u0014\n:\u000e\r , \u001b\n6 ) For each \u0001\u0010\u000f\n\nsub-model (3) is a good approximation of the full model (2). Since it is impossible to search\n\n\b\n . The weight\n\u0001\u0015\u0014 ) , such that the\n\nis the value of \u001f\n:\u000b\n\n\u0002\u0001\u0015\u0018\n\n\u000b\b(\n\n, we use the following greedy forward strategy:\n\n3.1 Basic algorithm\n\n\"\u0006\u0005\n\f\u0001\u000e\u0018-\n\u0010\n\n#&\u00019\u00038\u0005\b\u0001\u0015\r\u000f\u0005\n\n, let\n\n\n\u001f\n\u0012\u0011\u0012\u0013\u0012\n\n\u00038\u0005\b\u0001\u000e\r\u0006\u0005\n\n#&\n'\"\n\n\u0002\u0001\n\n!\u001a\u001b\n\n( .\n\n,90\n\n(\f\n(\f\n\n.\n\n($ \n\n\u0005\u0010\u0001\u0015\u0014 ) , \t\n\u0012\u0011\u0012\u0013\u0012\n\f\u00019\n\n\u000e\u0012\u0011\n\n\u0010\u0013\u0012\u0014\u0013\u0016\u0015\n\n\u0007\u000f\u0018\n\n\u0002\u0001\u0015\u0018\n\n\f\u0001\n\n\u0005\b\u0001\n\n\u000e\u0018\u0017\u0002\u0019\n\b(\u0010\b\u001b\u001a\u001d\u001c\u0016\u001e\u0004\n\b\u001b\u001a\u001d\u001c\u0016\u001e\u0004\n\n\f\u0001\u0015\u0018\n\n\u0002\u0001\n\n\b\n\u0010\n\n\b\n\n\u0003 , \u0001\u0015\u0018\n\u0005\b\u0001\n\n\f\u0001\u000e\n\n#&\u00019\u00038\u0005\b\u0001\u0015\r\u0019\u0005\n\u0012\u0011\u0012\u0013\u0012\n\u0003 , \u0001\n\n\u0001\u000e\u0014\u0016) ,\n\u0005\b\u0001\n\nto minimize\n\n\f\u0001\n\nFind \u001f\n\n(6)\n\n\u0018\u0003\u0002\n3#\u001f\n\u0007\u0016 \n\n(\u0003)\n\u0002\u0001\u0015\u0018\u0010\u0005\b\u0001\n) ; the regularization matrix \n= .\n\n#\u0011\u0001\n) ; E\n\u0014\u001b\u001a\n#\u0011\u0001\u0014\u000f\n\nwhere the regressor matrix \n\u0014\u001c\u001a\n(\f\u001e\u001d ) Let\nLet \u0014\n\n#\u0011\u0001\n\u0014\u001c\u001a\n(\f'& ) Repeat steps (\f\nWe call the points in \u0014\n\nimport points.\n\n:\u0018\u001b$#\n\n#\u0011\u0001\n) , \u001b\n\u000f\u0002\u001f\n\u000f%\u001f\n6 ) and (\f'\u001d ) until \u0019\n\n\u000f\u0002\u001f\n\n3.2 Revised algorithm\n\n\u0010\"!\n\nargmin\u000e \u0017\n\u0002\u0001\n) , \u0019\n\u0001 converges.\n\n\u0002\u0001\n\u000f\u0002\u001f\n\n , \t\n\n( .\n\n6 ) we need to use the\niteratively. When the number of import points E be-\n\nThe above algorithm is computationally feasible, but in step (\f\nNewton-Raphson method to \ufb01nd \u001f\ncomes large, the Newton-Raphson computation can be expensive. To reduce this computa-\ntion, we use a further approximation.\n\u0002\u0001\u0004\u0003 until it converges, we can just do a one-step iteration,\nInstead of iteratively computing \u001f\nand use it as an approximation to the converged one. To get a good approximation, we\nE , and use it as the initial value. This one-step update is similar to the score test in\ntake advantage of the \ufb01tted result from the current \u201coptimal\u201d \u0014\n, i.e., the sub-model when\n=\u0015:\nformula allows the weighted regression (5) to be computed in B\n6 ) for the basic algorithm:\nHence, we have the revised step (\f\n\ngeneralized linear models (GLM); but the latter does not have a penalty term. The updating\n\ntime.\n\n\u0002C\n\n\u001d\n:\n\n\u001d\n\u001d\n\u0014\n\n \n:\n0\n\n\u0015\n\n7\n\u0014\n:\n\u001d\n\u001d\n\u001f\n\u001d\n:\n\n\n \n\n+\n\u0003\n\n \n\u001f\n\b\n\u001d\n\u001d\n\b\n:\n\n\u001f\n\u001d\n+\n\u0003\n\u0003\n\b\n\u0005\n+\n\u0003\n\u0007\n3\n\u001f\n2\n+\n\u001d\n2\n3\n2\n7\n\u0014\n\u0017\n\u0014\n:\n:\n\u0004\n\u000f\n:\n\u0001\n\u001d\n\u0015\n\n\u0015\n\n\u001d\n\u0019\n\u000f\n\n:\n3\n\u0014\n\u0001\n\u0003\n0\n\u0004\n\u000f\n\n3\n(\n)\n\u0004\n\u000f\n7\n\b\n\n6\n\f\n\u0004\n\u000f\n\f\n\n\u000e\n\u000f\n:\n\n\n\u000f\n\"\n\u001f\n\u001d\n\u000f\n\n\b\n\u001f\n\n(\n\n\u000f\n\"\n\u001f\n\u001d\n\u000f\n\b\n\n6\n\u001f\n\u001d\n\u000f\n \n\n\u000f\n&\n\u001f\n\u001d\n\u000f\n\u000f\n\"\n:\n0\n\n\u0015\n\n7\n\u0014\n\u0017\n\n&\n\u0006\n\u0003\n!\n\u0001\n\u0015\n!\n\u000f\n\u000f\n&\n:\n0\n\n\u0015\n\u000f\n\n7\n\n&\n\u0006\n\u0003\n\u0003\n\u0017\n\n&\n\u0006\n\u0003\n\u0015\n\u000f\n!\n:\n=\n\u0014\n\u0001\n:\n\u0019\n\u000f\n\n\u0012\n:\n\u0001\n:\n\u0019\n:\n\t\n\b\n\u001d\n\u001d\n=\n\u0014\nE\n\n\f6\u0001 ) For each \u0001\n\n!\u0015\u001b\n\n\u0006\u0005\n\n& with a\n\n\u0001 with \u0019\n\u0005\b\u0006\n\t\f\u000b\r\u0004\n\u0005\u0007\u0006\u000e\u0004\n\n\" with a column, and \n\nin (5). Compute (6).\n\n, correspondingly augment \n\ncolumn and a row. Use the updating formula to \ufb01nd \u001f\n\nis a pre-chosen small integer, for example \u0003\n, for example, \u000f\n.\n\n\u0001 has converged. A\n\u0012\u0013\u0012\u0011\u0012 be the sequence\n, we compare \u0019\n\u0002 ,\n( . If the ratio \u0004\n\u0005\u0007\u0006\nis less\n%\u000f%'( , we stop adding new import\n\n(\f\n3.3 Stopping rule for adding point to \u0014\nIn step (\f'& ) of the basic algorithm, we need to decide whether \u0019\n\u00038\u0005\nnatural stopping rule is to look at the regularized NLL. Let \u0019\nof regularized NLL\u2019s obtained in step (\f\n& ). At each step \t\nwhere \u0003\nthan some pre-chosen small number \u000f\npoints to \u0014\n3.4 Choosing the regularization paramter \n\nSo far, we have assumed that the regularization parameter \n\nis \ufb01xed. In practice, we also\nneed to choose an \u201coptimal\u201d \n\n. We can randomly split all the data into a training set and a\ntuning set, and use the misclassi\ufb01cation error on the tuning set as a criterion for choosing\n. To reduce the computation, we take advantage of the fact that the regularized NLL\nconverges faster for a larger \n\n. Thus, instead of running the entire revised algorithm for\neach \n\n, we propose the following procedure, which combines both adding import points to\nand choosing the optimal \n :\n( ) Start with a large regularization parameter \n\n(\u0010\n#&\u0001\n6 ) Let \u0014\n( .\n\u0005\b\u0001\n:\u000e\r , \u001b\n\u0012\u0011\u0012\u0013\u0012\n(\u0010\n6\u0011 ), (\f\u001e\u001d ) and (\f\n\u001d ) Run steps (\f\n(\u0010\n#\u0011\u0001\nterion is satis\ufb01ed at \u0014\n& ) Decrease \n\n(\u0010\n(\u0010\u0013\u0012 ) Repeat steps (\u0010\nWe choose the optimal \n as the one that corresponds to the minimum misclassi\ufb01cation error\non the tuning set.\n\n) , \t\n& ) of the revised algorithm, until the stopping cri-\n\u0012\u0011\u0012\u0013\u0012\n\nmisclass\ufb01cation error on the tuning set.\n\n. Along the way, also compute the\n\n\u001d ) and (\u0010\n\n& ), starting with \u0014\n\nto a smaller value.\n\n:A#&\u0001\n\n\u0005\u0010\u0001\n\n\u0012\u0011\u0012\u0013\u0012\n\n) .\n\n.\n\n\u0005\u0010\u0001\n\n\u0005\b\u0001\n\n4 Simulation\n\nIn this section, we use a simulation to illustrate the IVM method. The data in each class\nare generated from a mixture of Gaussians (Hastie et al. (2001)). The simulation results\nare shown in Figure 2.\n\n4.1 Remarks\n\n( 3\n\nThe support points of the SVM are those which are close to the classi\ufb01cation boundary or\n\nIVM are those that decrease the regularized NLL the most, and can be either close to or\nfar from the classi\ufb01cation boundary. This difference is natural, because the SVM is only\n\n\f\u00019\n\u000b\n\u0002\u0001\u000e\n\u0010\n ]. The import points of the\nmisclassi\ufb01ed and usually have large weights [2\n(84\u0006687 , while the IVM also focuses on the\n\f\u00019\n\nconcerned with the classi\ufb01cation *\u0011+\n,\t.10\n\u0002\u0001\u000e\n . Though points away from the classi\ufb01cation boundary do not\nunknown probability 2\n\u0002\u0001\u000e\n . Figure 3 shows a comparison of the SVM and\nto estimating the unknown probability2\n\u0002C$D&\n , while the computational cost\n\n , where E\nis the number of import points. Since E does not\nthe IVM. The total computational cost of the SVM is B\nof the IVM method is B\n\ncontribute to determining the position of the classi\ufb01cation boundary, they may contribute\n\n&E8\n\n\u0002C\n\n\u000f\n\u001d\n\u0019\n\u0001\n+\n:\n+\n\u0004\n:\n%\n\u0012\n\n\u0014\n:\n\u0003\n\n\u0005\n\u0014\n:\n:\n\u0018\n\u0003\n\u0005\n\u0018\n&\n\u0006\n)\n\u0018\n\u0003\n\u0005\n\u0018\n&\n\u0006\n2\n2\n3\n\fRegularized NLL for different lambda\u2019s\n\nMisclassification rate for different lambda\u2019s\n\nRegularized NLL for the optimal lambda\n\n0\n5\n2\n\n0\n0\n2\n\n0\n5\n1\n\n0\n0\n1\n\n\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\n\u2022\u2022\n\u2022\u2022\n\n\u2022\u2022\u2022\u2022\u2022\u2022\u2022\n\n\u2022\u2022\n\n\u2022\u2022\u2022\u2022\n\n\u2022\u2022\n\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\n\u2022\u2022\u2022\u2022\u2022\n\u2022\u2022\u2022\n\u2022\u2022\u2022\u2022\u2022\u2022\u2022\n\u2022\u2022\u2022\u2022\n\u2022\u2022\u2022\u2022\n\u2022\u2022\u2022\u2022\n\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\n200\n\n100\n\n150\n\n50\n\n0\n\n4\n3\n.\n0\n\n2\n3\n.\n0\n\n0\n3\n.\n0\n\n8\n2\n.\n0\n\n6\n2\n.\n0\n\n4\n2\n.\n0\n\n2\n2\n.\n0\n\n\u2022\n\u2022\n\n\u2022\u2022\n\u2022\n\n\u2022\n\u2022\n\u2022\u2022\n\u2022\u2022\u2022\u2022\u2022\u2022\n\u2022\u2022\n\n\u2022\u2022\u2022\u2022\u2022\u2022\n\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\n\u2022\u2022\u2022\u2022\u2022\u2022\n\u2022\u2022\u2022\u2022\n\u2022\u2022\u2022\u2022\n\u2022\n\u2022\u2022\u2022\u2022\n\u2022\u2022\u2022\u2022\u2022\u2022\u2022\n\u2022\u2022\u2022\u2022\u2022\u2022\u2022\n\u2022\u2022\u2022\n\u2022\u2022\u2022\u2022\u2022\n\u2022\u2022\n\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\n\n0\n4\n2\n\n\u2022\n\n0\n2\n2\n\n0\n0\n2\n\n0\n8\n1\n\n0\n6\n1\n\n\u2022\n\n\u2022\n\n\u2022\u2022\n\u2022\n\u2022\n\u2022\n\u2022\n\n\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\n\n\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\u2022\n\n0\n\n50\n\n100\n\n150\n\n200\n\n0\n\n50\n\n100\n\n150\n\n200\n\nNo. of import points added\n\nNo. of import points added\n\nNo. of import points added\n\n\u0002\u0001\u0004\u0003\u0006\u0005\u0006\u0005 . The left and middle panels illustrate how to choose\n\u0007 , \u0007 decreases from \u000f\u0011\u0010\u0013\u0012\nto \u000f\u0011\u0014\u0015\u0010\u0013\u0012 . The minimum misclassi\ufb01cation\nis found to correspond to \u0007\u0019\u0001\u001a\u0005\u000e\r\n\u0007\u0018\u001b\u001f\u001d . The\n!\" \u001f\u0001#\u0003\u0006\u0007 .\n\n\u0007\u001c\u001b\u001e\u001d . The right panel is for the optimal \u0007\u0019\u0001\u001a\u0005\u0016\n\nFigure 2: Radial kernel is used.\n\n.\n\n\b\t\u0001\n\n\u0007 , \n\u000b\u0001\f\u0005\u000e\n\nthe optimal \u0007\n\u0005\u0006\u0005\nrate \u0005\u0016\r\n\u0003\u0006\u0007\u0018\u0017\nstopping criterion is satis\ufb01ed when \ntend to increase as C\n\nincreases, the computational cost of the IVM can be smaller than that\n\nof the SVM, especially for large training data sets.\n\n5 Multi-class case\n\nIn this section, we brie\ufb02y describe a generalization of the IVM to multi-class classi\ufb01cation.\n\n\u0007 , with each\nSuppose there are $\ncomponent being either 0 or 1, indicating which class the observation is in. Therefore \u0007\n(\u000f\u0005\n%'\u0005+%,)\n( th class as the basis, the\nindicates the response is in the $\n% .\nmulti-logit can be written as \u0004\u0019\u0003\n2/-\n\n( classes. We can write the response as an $\n-vector \u001f\nindicates the response is in the \t th class, and \u0007\n2.-\n\nHence the Bayes classi\ufb01cation rule is given by:\n\n\u0005&%('\n\n\u0005&%*)\n\n , \u0004\n\n(\u0019\u0005\n\n\u0012\u0013\u0012\u0011\u0012\n\n,\n\n(\u0019\u0005\n\n\u0012\u0013\u0012\u0011\u0012\n\n.\n\n\u000e\n\u000f\n\n\u000e\u0010\u000f\n, and\n\n\f\u0001\u000e\u0018-\n\u0010\n\n\u0003\u00180\n\n , \u0012\u0013\u0012\u0013\u0012\n\r1032323240\n\n\bA( th class. Using the $\n2.-\nargmax\u0001\nto index the classes, i.e. +\n/6:\n\u0012\u0013\u0012\u0013\u0012\n\n\b8797\u00067\n\r\u0019\u0002\u0001\u0015\u0018\n\n\u000e$\u000f\n\u00038\u0002\u0001\u0015\u0018\n\n\u0002\u0001\u0015\u0018\n\n1:\n\n/65\n\n(\u0003)\n\nto index the observations, %\n\nThen the regularized negative log-likelihood is\n\nWe use +\n\n(7)\n\n\u0007\u0019\u0018\u0004:\nwhere \u001f\n\n\f\u0007\u000f\u0018\n\n\u00035\u0005\u0010\u0007\u000f\u0018\n\n\f\u0001\u000e\u0018\n\u0005\u0010\u0007\u000f\u0018\n\n\u0007\u0016 \n\u0012\u0011\u0012\u0013\u0012\n\n\u0018\u0003\u0002\n\r\u0019\u0005\n\n,\n\nUsing the representer theorem (Kimeldorf et al. (1971)), the % th element of \u001f\nwhich minimizes \u0019\n(8)\n\nhas the form\n\n\f\u0001\u000e\n , \u0004\n\n\u0002\u0001\u000e\n ,\n\n\u0002\u0001\u000e\n\n\u0018\u0003\u0002\n\n\u0002\u0001\u0004\u0005\u0010\u0001\u0015\u0018\n\n\b\n\u0001\n:\n\u0007\n\u0015\n:\n%\n:\n\t\n$\n\u0015\n:\n$\n\b\n:\n(\n)\n\n2\n\u0003\n4\n\u0006\n\u0003\n\u0005\n\u0004\n-\n:\n(\n)\n\n4\n\u0006\n\u0003\n-\n\u0006\n\u0003\n:\n\u001b\n:\n\u0010\n\u0015\n-\n\u0006\n\u0003\n\u0019\n\u0004\n\u0001\n:\nC\n%\n:\n$\n\u0019\n:\n3\n\u0014\n\u0001\n\u0003\n0\n\u001f\n\u0018\n\u001f\n\u0004\n\n3\n\n(\n\b\n\"\n\n\u0003\n\b\n\"\n\n\u0003\n\n7\n\b\n\n6\n\f\n\u0004\n\f\n\n-\n\n \n\u001f\n\u0004\n\n\u0004\n\n\u0005\n\u0004\n\n\u0005\n\u0005\n\u0004\n-\n \n\f\n\u0004\n\f\n\n\u000e\n\u000f\n:\n-\n\u0001\n\u0015\n\u0002\n\u0003\n\f\n\u0004\n\u0015\n\f\n\n\u000e\n\u000f\n\u0004\n\u0015\n\u0004\n\u0015\n:\n\u0014\n\u0001\n\u0003\n\u001d\n\u0018\n\u0015\n\n\n\u0012\n\fSVM - with 107 support points\n\nIVM - with 21 import points\n\n\u2022\n\no\n\n\u2022\n\n\u2022\n\n\u2022\n\n\u2022\n\n\u2022\n\n\u2022\n\n\u2022\n\n\u2022\n\u2022\n\n\u2022\n\u2022\n\u2022\n\n\u2022\n\u2022\n\u2022\n\n\u2022\n\u2022\n\u2022\n\u2022\n\n. . . . . . . . . . . . . .\n++++++++++++++++++++++++++++++++++++++++++++++++++++\n+++\n\u2022\n. . . . . . . . . . . . .\n++++++++++++++++++++++++++++++++++++++++++++++++++++\n++++\n. . . . . . . . . . . . . .\n+++++++++++++++++++++++++++++++++++++++++++++++++++\n++++\n\u2022\n. . . . . . . . . . . . . .\n+++++++++++++++++++++++++++++++++++++++++++++++++++\n++++\n. . . . . . . . . . . . .\n+++++++++++++++++++++++++++++++++++++++++++++++++++\n+++++\n. . . . . . . . . . . . . .\n++++++++++++++++++++++++++++++++++++++++++++++++++\n+++++\n\u2022\n. . . . . . . . . . . . .\n++++++++++++++++++++++++++++++++++++++++++++++++++\n++++++\n. . . . . . . . . . . . .\n++++++++++++++++++++++++++++++++++++++++++++++++++\n++++++\no\n\u2022\n. . . . . . . . . . . . . .\no\n+++++++++++++++++++++++++++++++++++++++++++++++++\n++++++\n. . . . . . . . . . . . .\n+++++++++++++++++++++++++++++++++++++++++++++++++\n+++++++\n. . . . . . . . . . . . .\n\u2022\n+++++++++++++++++++++++++++++++++++++++++++++++++\n+++++++\n\u2022\n. . . . . . . . . . . . . .\n++++++++++++++++++++++++++++++++++++++++++++++++\n+++++++\n\u2022\n. . . . . . . . . . . . .\n++++++++++++++++++++++++++++++++++++++++++++++++\n++++++++\n. . . . . . . . . . . . .\n\u2022\n++++++++++++++++++++++++++++++++++++++++++++++++\n++++++++\n\u2022\n. . . . . . . . . . . . . .\n+++++++++++++++++++++++++++++++++++++++++++++++\n++++++++\n. . . . . . . . . . . . . .\n\u2022\n+++++++++++++++++++++++++++++++++++++++++++++++\n++++++++\n. . . . . . . . . . . . .\n+++++++++++++++++++++++++++++++++++++++++++++++\n+++++++++\n. . . . . . . . . . . . .\no\n+++++++++++++++++++++++++++++++++++++++++++++++\n+++++++++\n. . . . . . . . . . . . . .\n++++++++++++++++++++++++++++++++++++++++++++++\n+++++++++\no\n. . . . . . . . . . . . . .\n\u2022\n++++++++++++++++++++++++++++++++++++++++++++++\n+++++++++\no\n. . . . . . . . . . . . .\n\u2022\n++++++++++++++++++++++++++++++++++++++++++++++\n++++++++++\n. . . . . . . . . . . . .\n\u2022\n++++++++++++++++++++++++++++++++++++++++++++++\n++++++++++\n\u2022\n. . . . . . . . . . . . . .\n\u2022\n\u2022\n+++++++++++++++++++++++++++++++++++++++++++++\n++++++++++\n. . . . . . . . . . . . . .\n\u2022\n+++++++++++++++++++++++++++++++++++++++++++++\n++++++++++\n\u2022\n. . . . . . . . . . . . . .\n+++++++++++++++++++++++++++++++++++++++++++++\n++++++++++\n\u2022\no\n. . . . . . . . . . . . . .\n\u2022\no\n\u2022\u2022\n+++++++++++++++++++++++++++++++++++++++++++++\n++++++++++\no\n. . . . . . . . . . . . . .\n+++++++++++++++++++++++++++++++++++++++++++++\n++++++++++\n\u2022\n\u2022\n. . . . . . . . . . . . . .\n\u2022\n+++++++++++++++++++++++++++++++++++++++++++++\n++++++++++\n\u2022\n. . . . . . . . . . . . .\no\n\u2022\n+++++++++++++++++++++++++++++++++++++++++++++\n+++++++++++\n\u2022\no\n. . . . . . . . . . . . .\n\u2022\n+++++++++++++++++++++++++++++++++++++++++++++\n+++++++++++\n. . . . . . . . . . . . .\no\n+++++++++++++++++++++++++++++++++++++++++++++\n+++++++++++\n\u2022 \u2022\n. . . . . . . . . . . . .\n\u2022\n\u2022\n+++++++++++++++++++++++++++++++++++++++++++++\n+++++++++++\no\no\n\u2022\n. . . . . . . . . . . . .\n\u2022\n+++++++++++++++++++++++++++++++++++++++++++++\n+++++++++++\n\u2022\n\u2022\n. . . . . . . . . . . . .\n\u2022\no\n+++++++++++++++++++++++++++++++++++++++++++++\n+++++++++++\n\u2022\n. . . . . . . . . . . .\no\n\u2022\no\n++++++++++++++++++++++++++++++++++++++++++++++\n+++++++++++\n\u2022\n\u2022\u2022\n. . . . . . . . . . . .\no\n\u2022\n++++++++++++++++++++++++++++++++++++++++++++++\n+++++++++++\n\u2022\n\u2022\u2022\n\u2022\no\n. . . . . . . . . . . .\n++++++++++++++++++++++++++++++++++++++++++++++\n+++++++++++\n\u2022\n\u2022\n. . . . . . . . . . . .\n\u2022\no\n++++++++++++++++++++++++++++++++++++++++++++++\n+++++++++++\n\u2022\no\n\u2022\n. . . . . . . . . . . .\n++++++++++++++++++++++++++++++++++++++++++++++\n+++++++++++\no\n. . . .\n. . . . . . . . . . . .\no\n\u2022\no\n+++++++++++++++++++++++++++++\n++++++++++++++\n++++++++++\no\no\no\n. . . . . . .\n. . . . . . . . . . . .\n\u2022\n\u2022\no\n\u2022\n+++++++++++++\n+++++++++++++++++++++++++++\n++++++++++\n. . . . . . . .\n. . . . . . . . . . . .\n+++++++++++++\n++++++++++++++++++++++++++\n++++++++++\n\u2022\noo\n\u2022\n. . . . . . . . . .\n. . . . . . . . . . . .\n.\n++++++++++++\n+++++++++++++++++++++++++\n+++++++++\no\n. . . . . . . . . .\n. . . . . . . . . . . .\n.\n\u2022\n\u2022\no\n\u2022\n+++++++++++++++++++++++++\n+++++++++\n++++++++++++\no\no\n. . . . . . . . . . . .\n. . . . . . . . . . .\n.\n++++++++++++++++++++++++\n++++++++++++\n+++++++++\n\u2022\n. . . . . . . . . . .\n. . . . . . . . . . . . .\n.\n\u2022\n+++++++++++++++++++++++\n++++++++++++\n+++++++++\n. . . . . . . . . . . .\n. . . . . . . . . . . . . .\n. .\n++++++++++++++++++++++\n++++++++++++\n+++++++\no\n\u2022\no\n\u2022\n. . . . . . . . . . . . . .\n. . . . . . . . . . . .\n. .\no\n++++++++++++\n++++++++++++++++++++++\n+++++++\no\no\n. . . . . . . . . . . .\n. .\n. . . . . . . . . . . . . . . .\n\u2022\n+++++++++++++++++++++\n+++++++++++\n+++++++\n. . . . . . . . . . . .\n. . .\n. . . . . . . . . . . . . . . . .\no\n\u2022\n++++++++++++++++++++\n+++++++++++\n++++++\no\n. . . . . . . . . . . . .\n. . .\n. . . . . . . . . . . . . . . . . .\no\no\n\u2022\no\n+++++++++++++++++++\n+++++++++++\n+++++\no\n\u2022\no\n. . . . . . . . . . . . .\n. . . . . . . . . . . . . . . . . .\n. . . .\n+++++++++++++++++++\n+++++++++++\n++++\n\u2022\no\n. . . . . . . . . . . . .\n. . . .\n. . . . . . . . . . . . . . . . . . . .\n\u2022\n++++++++++++++++++\n++++++++++\n++++\n\u2022\n. . . . . . . . . . . . . .\n. . . . .\n. . . . . . . . . . . . . . . . . . . . .\n\u2022\no\n\u2022\n++\n+++++++++++++++++\n++++++++++\no\n. . . . . . . . . . . . . . . . . . . . . .\n. . . . . . . . . . . . . . . . . . . . .\n++++++++++\n++++++++++++++++\no\n. . . . . . . . . . . . . . . . . . . . .\n. . . . . . . . . . . . . . . . . . . . . . . .\no\n+++++++++++++++\n+++++++++\no\no\n. . . . . . . . . . . . . . . . . . . . . .\n. . . . . . . . . . . . . . . . . . . . . . . .\n+++++++++++++++\n++++++++\no\no\no\no\n. . . . . . . . . . . . . . . . . . . . . .\n. . . . . . . . . . . . . . . . . . . . . . . . . .\no\n+++++++\n++++++++++++++\no\n. . . . . . . . . . . . . . . . . . . . . . .\n. . . . . . . . . . . . . . . . . . . . . . . . . . . .\n+++++++++++++\n+++++\no\n. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .\n. . . . . . . . . . . . . . . . . . . . . . . .\no\n++++++++++++\n+++\noo\n\u2022\n. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .\n++++++++++++\n. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .\n+++++++++++\n. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .\n\u2022\n++++++++++\no\n\u2022\no\n. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .\n++++++++++\no\n. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .\n+++++++++\no\n. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .\n+++++++++\n. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .\n++++++++\no\no\no\n. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .\n++++++++\n. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .\n+++++++\n. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .\n+++++++\n. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .\n++++++\n. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .\n++++++\n. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .\no\no\n+++++\n. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .\no\n. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .\n. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .\no\no\no\n. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .\n. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .\no\n. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .\no\n. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .\no\no\n. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .\no\n. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .\n. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .\n. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .\n. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .\n. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .\n. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .\n. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .\n. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .\n. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .\n. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .\n. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .\n. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .\n. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .\n. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .\n. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .\n. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .\n. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .\n. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .\n\nTraining Error: 0.160\nTest Error: 0.218\nBayes Error: 0.210\n\n++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n\n+++++\n+++++\n++++++\n++++++\n+++++++\n+++++++\n\no\no\no\no\n\no\no\no\n\no\no\n\no\no\n\no\no\n\n\u2022\n\n\u2022\n\no\n\no\n\no\n\no\n\no\n\no\n\n\u2022\n\no\n\n\u2022\n\n\u2022\n\n\u2022\n\n\u2022\n\n\u2022\n\n\u2022\n\n\u2022\n\n\u2022\n\u2022\n\n\u2022\n\u2022\n\u2022\n\n\u2022\n\u2022\n\u2022\n\n\u2022\n\u2022\n\u2022\n\u2022\n\n. . . . . . . . . . . .\n++++++++++++++++++++++++++++++++++++++++++++++\n+++++++++++\n\u2022\n. . . . . . . . . . . .\n++++++++++++++++++++++++++++++++++++++++++++++\n+++++++++++\n. . . . . . . . . . . . .\n++++++++++++++++++++++++++++++++++++++++++++++\n++++++++++\n\u2022\n. . . . . . . . . . . . . .\n+++++++++++++++++++++++++++++++++++++++++++++\n++++++++++\n. . . . . . . . . . . . . .\n+++++++++++++++++++++++++++++++++++++++++++++\n++++++++++\n. . . . . . . . . . . . . .\n+++++++++++++++++++++++++++++++++++++++++++++\n++++++++++\n\u2022\n. . . . . . . . . . . . . . .\n++++++++++++++++++++++++++++++++++++++++++++\n++++++++++\n. . . . . . . . . . . . . . .\n++++++++++++++++++++++++++++++++++++++++++++\n++++++++++\no\n\u2022\n. . . . . . . . . . . . . . .\no\n++++++++++++++++++++++++++++++++++++++++++++\n++++++++++\n. . . . . . . . . . . . . . .\n++++++++++++++++++++++++++++++++++++++++++++\n++++++++++\n. . . . . . . . . . . . . . .\n\u2022\n++++++++++++++++++++++++++++++++++++++++++++\n++++++++++\n\u2022\n. . . . . . . . . . . . . . .\n++++++++++++++++++++++++++++++++++++++++++++\n++++++++++\n\u2022\n. . . . . . . . . . . . . . .\n++++++++++++++++++++++++++++++++++++++++++++\n++++++++++\n. . . . . . . . . . . . . . .\n\u2022\n++++++++++++++++++++++++++++++++++++++++++++\n++++++++++\n\u2022\n. . . . . . . . . . . . . . .\n++++++++++++++++++++++++++++++++++++++++++++\n++++++++++\n. . . . . . . . . . . . . . .\n\u2022\n++++++++++++++++++++++++++++++++++++++++++++\n++++++++++\n. . . . . . . . . . . . . . .\n++++++++++++++++++++++++++++++++++++++++++++\n++++++++++\n. . . . . . . . . . . . . . .\no\n++++++++++++++++++++++++++++++++++++++++++++\n++++++++++\n. . . . . . . . . . . . . . .\n++++++++++++++++++++++++++++++++++++++++++++\n++++++++++\no\n. . . . . . . . . . . . . . .\n\u2022\n++++++++++++++++++++++++++++++++++++++++++++\n++++++++++\no\n. . . . . . . . . . . . . . .\n\u2022\n++++++++++++++++++++++++++++++++++++++++++++\n++++++++++\n. . . . . . . . . . . . . .\n\u2022\n++++++++++++++++++++++++++++++++++++++++++++\n+++++++++++\n\u2022\n. . . . . . . . . . . . . .\n\u2022\n\u2022\n++++++++++++++++++++++++++++++++++++++++++++\n+++++++++++\n. . . . . . . . . . . . . .\n\u2022\n++++++++++++++++++++++++++++++++++++++++++++\n+++++++++++\n\u2022\n. . . . . . . . . . . . . .\n++++++++++++++++++++++++++++++++++++++++++++\n+++++++++++\n\u2022\no\n. . . . . . . . . . . . . .\n\u2022\no\n\u2022\u2022\n++++++++++++++++++++++++++++++++++++++++++++\n+++++++++++\no\n. . . . . . . . . . . . . .\n++++++++++++++++++++++++++++++++++++++++++++\n+++++++++++\n\u2022\n\u2022\n. . . . . . . . . . . . . .\n\u2022\n++++++++++++++++++++++++++++++++++++++++++++\n+++++++++++\n\u2022\n. . . . . . . . . . . . . .\no\n++++++++++++++++++++++++++++++++++++++++++++\n+++++++++++\n\u2022\no\n. . . . . . . . . . . . . .\n.\n\u2022\n++++++++++++++++++++++++++++++++++++++++++++\n++++++++++\n. . . . . . . . . . . . .\n.\no\n+++++++++++++++++++++++++++++++++++++++++++++\n++++++++++\n\u2022 \u2022\n. . . . . . . . . . . . .\n.\n\u2022\n\u2022\n+++++++++++++++++++++++++++++++++++++++++++++\n++++++++++\no\no\n. . . . . . . . . . . . .\n.\n\u2022\n+++++++++++++++++++++++++++++++++++++++++++++\n++++++++++\n\u2022\n\u2022\n.\n. . . . . . . . . . . . .\n\u2022\no\n+++++++++++++++++++++++++++++++++++++++++++++\n++++++++++\n\u2022\n.\n. . . . . . . . . . . . .\no\n\u2022\no\n+++++++++++++++++++++++++++++++++++++++++++++\n++++++++++\n\u2022\n\u2022\u2022\n.\n. . . . . . . . . . . .\no\n\u2022\n++++++++++++++++++++++++++++++++++++++++++++++\n++++++++++\n\u2022\n\u2022\no\n.\n. . . . . . . . . . . . .\n+++++++++\n++++++++++++++++++++++++++++++++++++++++++++++\n\u2022\n\u2022\n. .\n. . . . . . . . . . . . .\n\u2022\no\n++++++++++++++++++++++++++++++++++++++++++++++\n++++++++\n\u2022\no\n\u2022\n.\n. . . . . . . . . . . . .\n. .\n+++++++++++++++++++++++++++++ ++++++++++++++++\n++++++++\no\n. . . . .\n. .\n. . . . . . . . . . . .\no\n\u2022\no\n+++++++++++++++\n+++++++++++++++++++++++++++\n++++++++\no\no\no\n. . . . . . .\n. .\n. . . . . . . . . . . .\n\u2022\n\u2022\no\n\u2022\n++++++++++++++\n++++++++++++++++++++++++++\n++++++++\n. . . . . . . . .\n. . . . . . . . . . . .\n. .\n+++++++++++++\n+++++++++++++++++++++++++\n++++++++\n\u2022\noo\n\u2022\n. . . . . . . . .\n. . . . . . . . . . . .\n. .\n+++++++++++++++++++++++++\n+++++++++++++\n++++++++\no\n. . . . . . . . . . .\n. . . . . . . . . . . .\n. .\n\u2022\n\u2022\no\n\u2022\n++++++++++++\n++++++++++++++++++++++++\n++++++++\no\no\n. . . . . . . . . . .\n. . . . . . . . . . .\n. .\n+++++++++++++\n++++++++++++++++++++++++\n++++++++\n\u2022\n. . . . . . . . . . .\n. . . . . . . . . . . .\n. . .\n\u2022\n+++++++++++++\n+++++++++++++++++++++++\n+++++++\n. . . . . . . . . . .\n. . . . . . . . . . . . . .\n. . .\n++++++++++++\n++++++++++++++++++++++\n+++++++\no\n\u2022\no\n\u2022\n. . . . . . . . . . . . . .\n. . .\n. . . . . . . . . . . .\no\n++++++++++++\n++++++++++++++++++++++\n++++++\no\n. . . . . . . . . . . . . . .\n. . .\n. . . . . . . . . . . .\n\u2022\n++++++++++++\n+++++++++++++++++++++\n++++++\n. . . . . . . . . . . . . . .\n. . .\n. . . . . . . . . . . .\no\n\u2022\n++++++++++++\n+++++++++++++++++++++\n++++++\no\n. . . . . . . . . . . . . . . . .\n. . .\n. . . . . . . . . . . .\no\no\n\u2022\no\n++++++++++++++++++++\n+++++++++++\n++++++\no\n\u2022\no\n. . . . . . . . . . . .\n. . . . . . . . . . . . . . . . . .\n. . .\n+++++++++++\n+++++++++++++++++++\n++++++\n\u2022\no\n. . . . . . . . . . . . . . . . . .\n. . . .\n. . . . . . . . . . . . .\n\u2022\n+++++++++++\n+++++++++++++++++++\n\u2022\n. . . .\n. . . . . . . . . . . . .\n. . . . . . . . . . . . . . . . . . . .\n\u2022\no\n\u2022\n++++++++++\n++++++++++++++++++\no\n. . . .\n. . . . . . . . . . . . .\n. . . . . . . . . . . . . . . . . . . . .\n++++++++++\n+++++++++++++++++\no\n. . . . . . . . . . . . . . . . . . . . . .\n. . . . . . . . . . . . . .\n. . . .\no\n++++++++++\n++++++++++++++++\no\no\n. . . . .\n. . . . . . . . . . . . . . .\n. . . . . . . . . . . . . . . . . . . . . . .\n++++++++\n++++++++++++++++\no\no\no\no\n. . . . . . . . . . . . . . . . . . . . . .\n. . . . . . . . . . . . . . . . . . . . . . . .\no\n++++++++\n+++++++++++++++\no\n. . . . . . . . . . . . . . . . . . . . . . .\n. . . . . . . . . . . . . . . . . . . . . . . . . .\n++++++\n++++++++++++++\no\n. . . . . . . . . . . . . . . . . . . . . . . . .\n. . . . . . . . . . . . . . . . . . . . . . . . . . . .\no\n++\n++++++++++++++\noo\n\u2022\n. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .\n+++++++++++++\n. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .\n+++++++++++++\n. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .\n\u2022\n++++++++++++\no\n\u2022\no\n. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .\n++++++++++++\no\n. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .\n+++++++++++\no\n. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .\n+++++++++++\n. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .\n+++++++++++\no\no\no\n. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .\n++++++++++\n. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .\n++++++++++\n+++++++++\n+\n+++++++++\n++\n+++++++++\n++\no\no\n++++++++\n++\no\n++++++++\n++\n++++++++\n++\no\no\n++++++++\n++\no\n++++++++\n++\n++++++++\n++\no\n+++++++\n++\no\no\n+++++++\n++\no\n+++++++\n++\no\n++++++++\n++\n++++++++\n++\n++++++++\n+\n++++++++\n+\n++++++++\n+\n++++++++\n+\n+++++++++\n+\n+++++++++\n+\n+++++++++\n+\n++++++++++\n+\n++++++++++\n+\n+++++++++++\n+\n++++++++++++\n+\n++++++++++++\n+\n+++++++++++++\n+\n++++++++++++++\n+\n+++++++++++++++\n+\n++++++++++++++++\n+\n\n. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .\n. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .\n. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .\n. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .\n. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .\n. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .\n. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .\n. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .\n. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .\n. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .\n. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .\n. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .\n. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .\n. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .\n. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .\n. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .\n. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .\n. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .\n. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .\n. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .\n. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .\n. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .\n. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .\n. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .\n. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .\n. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .\n. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .\n. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .\n. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .\n. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .\n\nTraining Error: 0.150\nTest Error: 0.219\nBayes Error: 0.210\n\n\u2022\n\u2022\n\u2022\u2022\no\n++++\n++++\n++++\n+++\n++\n\no\no\no\no\n\no\no\no\n\no\no\n\no\no\n\no\no\n\n\u2022\n\n\u2022\n\no\n\no\n\no\n\no\n\no\n\no\n\nFigure 3: The solid lines are the classi\ufb01cation boundaries; the dotted lines are the Bayes rule\nboundaries. For the SVM, the dashed lines are the edges of the margin. For the IVM, the dashed lines\n\nare the\u0002\u0001\u0004\u0003\u0006\u0005\n\n\u0001(\u0005\u000e\n\n\u0003\u0006\u001d and \u0005\u0016\n\n\u0007\u0006\u001d\n\nlines.\n\nHence, (7) becomes\n\n(9)\n\n\u0019\u000b\n\n\u0003\r\f\n\n\u0003\u000f\u000e\n\n(\u0003)\n\n\u000b\u001f\n\n\b(\u0010\b\n\n\t\b\n\n , '\" and '& are de\ufb01ned in the same way as in the binary\n\n . Figure 4 is a simulation of the multi-class IVM. The data in each class are\n\n\u0012\u0013\u0012\u0013\u0012\nis the + th row of \n\n\" .\n\nThe multi-class IVM procedure is similar to the binary case, and the computational cost is\n\nwhere \b\ncase; and \n\r\u0011E8\n\ngenerated from a mixture of Gaussians (Hastie et al. (2001)).\n\nMulti-class IVM - with 32 import points\n\no\n\no\n\no\n\no\no\n\n+++++++++++++\n\n++++++++++++++++++++++\n\nxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\nxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\nxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\nxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\nxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\nxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\nxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\nxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\nxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\nxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\nxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\nxxxxxx\nxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\nxx\n+++++++++++++++++++++++++++\nxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\n+++++++++++++++++++++++++++++\nxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\n+++++++++++++++++++++++++++++++\nxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\n+++++++++++++++++++++++++++++++++\nxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\n++++++++++++++++++++++++++++++++++\nxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\n++++++++++++++++++++++++++++++++++++\nxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\n+++++++++++++++++++++++++++++++++++++\nxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\n++++++++++++++++++++++++++++++++++++++\nxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\no\n++++++++++++++++++++++++++++++++++++++++\nxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\no\no\n+++++++++++++++++++++++++++++++++++++++++\nxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\no\no\n++++++++++++++++++++++++++++++++++++++++++\nxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\n+++++++++++++++++++++++++++++++++++++++++++\nxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\no\n+++++++++++++++++++++++++++++++++++++++++++++\nxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\no\no\n+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\nxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\no\no\n+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\nxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\nx\no\no\n++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\nxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\nxxxx\no\no\no\no\no\n++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\nxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\nxxxxxxxx\no\no o\no\no\n+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\nxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\nxxxxxxxxxxxxxx\no\no\no\n+++++++++++++++++++++++++++++++++++++++++++++++++++++\nxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\nxxxxxxxxxxxxxxxxxxxxxxx\no\no\no\n++++++++++++++++++++++++++++++++++++++++++++\nxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\nxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\no\n++++++++++++++++++++++++++++++++++++++++\nxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\nxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\no\no\no\no\noo\no\n++++++++++++++++++++++++++++++++++++++++\nxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\nxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\no\no\no\no\no\n+++++++++++++++++++++++++++++++++++++++\nxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\nxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\no\n++++++++++++++++++++++++++++++++++++++\nxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\nxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\no\n++++++++++++++++++++++++++++++++++++++\nxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\nxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\no\n++++++++++++++++++++++++++++++++++++++\nxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\nxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\no\no\no\no\no\n++++++++++++++++++++++++++++++++++++++\nxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\nxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\no\no\no\no\n......\n...............\noo\no\n++++++++++++++++++++++++++++++++++++\nxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\n..........................................\n++++++++++++++++++++++++++++++++++\nxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\no\n.......\n...........................................\n++++++++++++++++++++++++++++\no\nxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\no\n............................................\n..............\no\n+++++++++++++++++++++++\nxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\no\n...................\n.............................................\n++++++++++++++++++++\nxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\n..............................................\n.......................\n++++++++++++++++++\nxxxxxxxxxxxxxxxxxxxxxxxxxxx\no\no\no\n...............................................\n..........................\n+++++++++++++++++\nxxxxxxxxxxxxxxxxxxxxxxxx\no\no\n...............................................\n.............................\no\n+++++++++++++++++\no\nxxxxxxxxxxxxxxxxxxxxx\n................................................\n................................\no\n+++++++++++++++++\nxxxxxxxxxxxxxxxxx\no\n................................................\n....................................\n++++++++++++++++++\nxxxxxxxxxxxxxxxx\no\n.................................................\n...........................................\n++++++++++++++++++\no\n..............................................\n.................................................\n+++++++++++++++++++\no\no\n..............................................\n..................................................\n++++++++++++++++++\n..............................................\n..................................................\no\no\n++++++++++++++++++\no\n...............................................\n....................................................\no\n+++++++++++++++\n......................................................\n.................................................\n+++++++++++\n..................................................................................................................\n..................................................................................................................\n..................................................................................................................\n..................................................................................................................\n..................................................................................................................\n..................................................................................................................\n..................................................................................................................\n..................................................................................................................\n..................................................................................................................\n..................................................................................................................\n..................................................................................................................\n..................................................................................................................\n..................................................................................................................\nTraining Error: 0.237\n..................................................................................................................\n..................................................................................................................\n..................................................................................................................\n..................................................................................................................\nTest Error: 0.259\n..................................................................................................................\n..................................................................................................................\n..................................................................................................................\nBayes Error: 0.251\n..................................................................................................................\n\no\no\no\no\no\no\no\no\no\no\no\n\no\nxxxxxxxxxxxxxxxxxxx\n\no\no\n\no\n\no\n\no\n\nFigure 4: Radial kernel is used. \u0010\u0012\u0011\n\n\u0001(\u001b , \f\u0001(\u001b\u001f\u0005\u0006\u0005 , \u0007\n\n\u0001(\u0005\u0016\n\n\u001b\u0014\u0013\u0016\u0015 , \n\n \u0006\u0001#\u001b\u001f\u0003 .\n\n\u0019\n:\n3\n\u0014\n\u0001\n\u0018\n\u0002\n\u0003\n0\n\u001f\n\u0007\n \n\u0018\n\n\n\"\n\n+\n\u0005\n\n \n3\n\u001f\n(\n \n\"\n\n\n\u0018\n0\n\n7\n\b\n\n6\n-\n\u0001\n\u0015\n\u0002\n\u0003\n\u001f\n\u001d\n \n\u0015\n\n&\n\u001f\n\u001d\n\u0015\n:\n\u001d\n\u0003\n\u001f\n\u001d\n-\n\n:\n\n\u001d\n\u0018\n\u0015\n\"\n\n+\n\u0005\n\nB\n\n$\nC\n\u0007\n!\n\f6 Conclusion\n\nWe have discussed the import vector machine (IVM) method in both binary and multi-class\nclassi\ufb01cation. We showed that it not only performs as well as the SVM, but also provides\nfor\nis the number of import\n\n\f\u00019\n . The computational cost of the IVM is B\n\n for the multi-class case, where E\n\n\u0011E8\n\n\u0011E8\r\nan estimate of the probability 2\nthe binary case and B\n\npoints.\n\nAcknowledgments\n\nWe thank Dylan Small, John Storey, Rob Tibshirani, and Jingming Yan for their helpful\ncomments. Ji Zhu is partially supported by the Stanford Graduate Fellowship. Trevor\nHastie is partially supported by grant DMS-9803645 from the National Science Founda-\ntion, and grant ROI-CA-72028-01 from the National Institutes of Health. Thanks to Grace\nWahba and Chris Williams for pointing out several interesting and important references.\nWe also want to thank the anonymous NIPS referees who helped improve this paper.\n\nReferences\n\n[1] Burges, C.J.C. (1998) A tutorial on support vector machines for pattern recognition. In Data\nMining and Knowledge Discovery. Kluwer Academic Publishers, Boston. (Volume 2)\n\n[2] Evgeniou, T., Pontil, M., & Poggio., T. (1999) Regularization networks and support vector ma-\nchines. In A.J. Smola, P. Bartlett, B. Sch\u00a8olkopf, and C. Schuurmans, editors, Advances in Large\nMargin Classi\ufb01ers. MIT Press.\n\n[3] Green, P. & Yandell, B. (1985) Semi-parametric generalized linear models. Proceedings 2nd\nInternational GLIM Conference, Lancaster, Lecture notes in Statistics No. 32 44-55 Springer-Verlag,\nNew York.\n\n[4] Hastie, T. & Tibshirani, R. (1990) Generalized Additive Models, Chapman and Hall.\n\n[5] Hastie, T., Tibshirani, R., & Friedman, J.(2001) The elements of statistical learning. In print.\n\n[6] Lin, X., Wahba, G., Xiang, D., Gao, F., Klein, R. & Klein B. (1998), Smoothing spline ANOVA\nmodels for large data sets with Bernoulli observations and the randomized GACV. Technical Report\n998, Department of Statistics, University of Wisconsin, Madison WI.\n\n[7] Kimeldorf, G. & Wahba, G. (1971) Some results on Tchebychef\ufb01an spline functions. J. Math.\nAnal. Applic. 33, 82-95.\n\n[8] Smola, A. & Sch\u00a8olkopf, B. (2000) Sparse Greedy Matrix Approximation for Machine Learning.\nIn Proceedings of the Seventeenth International Conference on Machine Learning. Morgan Kauf-\nmann Publishers.\n\n[9] Wahba, G. (1998) Support Vector Machine, Reproducing Kernel Hilbert Spaces and the Ran-\ndomized GACV. Technical Report 984rr, Department of Statistics, University of Wisconsin, Madison\nWI.\n\n[10] Wahba, G., Gu, C., Wang, Y., & Chappell, R. (1995) Soft Classi\ufb01cation, a.k.a. Risk Estima-\ntion, via Penalized Log Likelihood and Smoothing Spline Analysis of Variance. In D.H. Wolpert,\neditor, The Mathematics of Generalization. Santa Fe Institute Studies in the Sciences of Complexity.\nAddison-Wesley Publisher.\n\n[11] Williams, C. & Seeger, M (2001) Using the Nystrom Method to Speed Up Kernel Machines.\nIn T. K. Leen, T. G. Diettrich, and V. Tresp, editors, Advances in Neural Information Processing\nSystems 13. MIT Press.\n\n\nC\n\n\n$\nC\n\f", "award": [], "sourceid": 2059, "authors": [{"given_name": "Ji", "family_name": "Zhu", "institution": null}, {"given_name": "Trevor", "family_name": "Hastie", "institution": null}]}