{"title": "Diverse Ensemble Evolution: Curriculum Data-Model Marriage", "book": "Advances in Neural Information Processing Systems", "page_first": 5905, "page_last": 5916, "abstract": "We study a new method (``Diverse Ensemble Evolution (DivE$^2$)'') to train an ensemble of machine learning models that assigns data to models at each training epoch based on each model's current expertise and an intra- and inter-model diversity reward. DivE$^2$ schedules, over the course of training epochs, the relative importance of these characteristics; it starts by selecting easy samples for each model, and then gradually adjusts towards the models having specialized and complementary expertise on subsets of the training data, thereby encouraging high accuracy of the ensemble. We utilize an intra-model diversity term on data assigned to each model, and an inter-model diversity term on data assigned to pairs of models, to penalize both within-model and cross-model redundancy. We formulate the data-model marriage problem as a generalized bipartite matching, represented as submodular maximization subject to two matroid constraints. DivE$^2$ solves a sequence of continuous-combinatorial optimizations with slowly varying objectives and constraints. The combinatorial part handles the data-model marriage while the continuous part updates model parameters based on the assignments. In experiments, DivE$^2$ outperforms other ensemble training methods under a variety of model aggregation techniques, while also maintaining competitive efficiency.", "full_text": "Diverse Ensemble Evolution:\n\nCurriculum Data-Model Marriage\n\nDepts. of Computer Science and Engineering, and Electrical and Computer Engineering\n\nTianyi Zhou, Shengjie Wang, Jeff A. Bilmes\n\nUniversity of Washington, Seattle\n\n{tianyizh, wangsj, bilmes}@uw.edu\n\nAbstract\n\nWe study a new method \u201cDiverse Ensemble Evolution (DivE2)\u201d to train an en-\nsemble of machine learning models that assigns data to models at each training\nepoch based on each model\u2019s current expertise and an intra- and inter-model di-\nversity reward. DivE2 schedules, over the course of training epochs, the relative\nimportance of these characteristics; it starts by selecting easy samples for each\nmodel, and then gradually adjusts towards the models having specialized and\ncomplementary expertise on subsets of the training data, thereby encouraging\nhigh accuracy of the ensemble. We utilize an intra-model diversity term on data\nassigned to each model, and an inter-model diversity term on data assigned to\npairs of models, to penalize both within-model and cross-model redundancy. We\nformulate the data-model marriage problem as a generalized bipartite matching,\nrepresented as submodular maximization subject to two matroid constraints. DivE2\nsolves a sequence of continuous-combinatorial optimizations with slowly varying\nobjectives and constraints. The combinatorial part handles the data-model marriage\nwhile the continuous part updates model parameters based on the assignments. In\nexperiments, DivE2 outperforms other ensemble training methods under a variety\nof model aggregation techniques, while also maintaining competitive ef\ufb01ciency.\n\n1\n\nIntroduction\n\nEnsemble methods [7, 57, 31, 8] are simple and powerful machine learning approaches to obtain\nimproved performance by aggregating predictions (e.g., majority voting or weighted averaging) over\nmultiple models. Over the past few decades, they have been widely applied, consistently yielding\ngood results. For neural networks (NN) in particular, ensemble methods have shown their utility\nfrom the early 1980s [72, 28, 39, 10] to recent times [50, 27, 66, 20]. State-of-the-art results on many\ncontemporary competitions/benchmarks are achieved via ensembles of deep neural networks (DNNs),\ne.g., ImageNet [17], SQuAD [55], and the Kaggle competitions (https://www.kaggle.com/).\nIn addition to boosting state-of-the-art performance of collections of large models, ensembles of\nsmall and weak models can achieve performance comparable to much larger individual models,\nand this can be useful when machine resources are limited. Inference over an ensemble of models,\nmoreover, can be easily parallelized even on a distributed machine.\nA key reason for the success of ensemble methods is that the diversity among different models can\nreduce the variance of the combined predictions and improve generalization. Intuitively, diverse\nmodels tend to make mistakes on different samples in different ways (e.g., assigning largest probability\nto different wrong classes), so during majority voting or averaging, those different mistakes cancel\neach other out and the correct predictions can prevail. As neural networks grow larger in size and\nintricacy, their variance correspondingly increases, offering opportunity for reduction by a diverse\nensemble of such networks.\n\n32nd Conference on Neural Information Processing Systems (NeurIPS 2018), Montr\u00e9al, Canada.\n\n\fRandomization is a widely-used technique to produce diverse ensembles. Classical ensemble methods\nsuch as random initialization [17, 63], random forests [31, 8] and Bagging [7, 19], encourage diversity\nby randomly initializing starting points/subspaces or resampling the training set for different models.\nEnsemble-like methods for DNNs, e.g., dropout [61] and swapout [59], implicitly train multiple\ndiverse models by randomly dropping hidden units out during the training of a single model. Diversity\ncan also be promoted by sequentially training multiple models to encourage a difference between the\ncurrent and previously trained models, such as Boosting [57, 23, 50] and snapshot ensembles [32].\nSuch sequential methods, however, are hard to parallelize and can lead to long training times when\napplied to neural networks.\nDespite the consensus that diversity is essential to ensemble training, there is little work explicitly\nencouraging and controlling diversity during ensemble model training. Most previous methods encour-\nage diversity only implicitly, and are incapable of adjusting the amount of diversity precisely based\non criterion determined during different learning stages, nor do they have an explicit diversity repre-\nsentation. Some methods implicitly encourage diversity during training, but they rely on learning rate\nscheduling (e.g., snapshot ensembles [32]) or end-to-end training of an additive combination of mod-\nels (e.g., mixture of experts [33, 34, 58]) to promote diversity, which is hard to control and interpret.\nMoreover, many existing ensemble training methods train all models in the ensemble on all samples\nin the training set by repeatedly iterating through it, so the training cost increases linearly with\nthe number of models and number of samples. In such case, each model might waste much of its\ntime on a large number of redundant or irrelevant samples that have already been learnt, and that\nmight contribute nearly zero-valued gradients. The performance of an ensemble on each sample only\ndepends on whether a subset of models (e.g., half for majority voting) makes a correct prediction,\nso it should be unnecessary to train each model on every sample.\nIn this paper, we aim to achieve an ensemble of models using explicitly encouraged diversity and\nfocused expertise, i.e., each model is an expert in a suf\ufb01ciently large local-region of the data space,\nand all the models together cover the entire space. We propose an ef\ufb01cient meta-algorithm \u201cdiverse\nensemble evolution (DivE2)\u201d, that \u201cevolves\u201d the ensemble adaptively by changing over stages both\nthe diversity encouragement and each model\u2019s expertise, and this is based on information available\nduring ensemble training. It does this encouraging both intra- and inter-model diversity. Each training\nstage is formulated as a hybrid continuous-combinatorial optimization. The combinatorial part solves\na data-model-marriage assignment via submodular generalized bipartite matchings; the algorithm\nexplicitly controls the diversity of the ensemble and the expertise of each model by assigning different\nsubsets of the training data to different models. The continuous part trains each model\u2019s parameters\nusing the assigned subset of data. At each stage, all the models may be updated in parallel after\nreceiving their assigned data.\nA similar approach to encourage inter-model diversity was used in[15] but there diversity of different\nmodels is achieved by encouraging diverse subsets of features and the goal was to cluster the features\ninto potentially overlapping groups; here we are encouraging diverse subsets of samples to be assigned\nto models during the training process and we are matching data samples to models.\nWe apply DivE2 to four benchmark datasets, and show that it improves over randomization-based\nensemble training methods on a variety of approaches to aggregate ensemble models into a single\nprediction. Moreover, with model selection based ensemble aggregation (de\ufb01ned below), DivE2 can\nquickly reach reasonably good ensemble performance after only a few learning stages even though\neach individual model has poor performance on the entire training set. Furthermore, DivE2 exhibits\ncompetitive ef\ufb01ciency and good of model expertise interpretability, both of which can be important\nin DNN training.\n2 Diverse Ensemble Evolution (DivE2): Formulation\n2.1 Data-Model Marriage\nAn ensemble of models can make an accurate prediction on a sample without requiring each model\nmaking accurate predictions on that sample [41, 26, 42]. Rather, it requires a subset of models to\nproduce accurate predictions, and the remainder may err in different ways. Hence, rather than training\neach model on the entire training set, we may in theory assign a data subset to each model. Then,\neach sample is learned by a subset of models, and different models are trained on different subsets\nthereby avoiding common mistakes across models.\nConsider a weighted bipartite graph (see Fig. 1), with the set of n = |V | training samples V on the\nleft side, the set of m = |U| models U on the right side, and edges E , {(vj, ui)|vj 2 V, ui 2 U}\n\n2\n\n\f(1)\n\nmax\n\ni=1\n\nSamples\n\n( `(vj; wi)),\n\n\u2264\"(k=2)\n\nconnecting all sample-model pairs with edge weights de\ufb01ned by the loss `(vj; wi) of sample vj =\n(xj, yj) (where xj is the features and yj is the label(s)) on model ui (where model ui is parameterized\nby wi). We wish to marry samples with models by selecting a subset of edges having overall small\nloss. We can express this as follows:\nmax\n{wi}m\n\nA\u2713E X(vj ,ui)2A\nwhere `(vj; wi) translates loss to reward (or accuracy), and is a constant larger than any\nper-sample loss on any model, i.e., `(vj; wi),8i, j 1.\nModels\nWith no constraints, all edges are selected thus requiring all models to\nlearn all samples. As mentioned above, for ensembles, every sample\nneed only be learned by a few models, and thus, for any sample v, we\nmay wish to limit the number of incident edges selected to be no greater\nthan k. This can be achieved using partition matroid MV = (E,IV ),\nwhere IV = (I1, I2, . . . , In) and Ii \u2713 E. IV contains all subsets of\nE where no sample is incident to more than k edges in any subset, i.e.\nIV = {A \u2713 E : |A \\ (v)|\uf8ff k,8v 2 V }, where (v) \u2713 E is the set\nof edges incident to v (likewise for (u), u 2 U). Therefore, as long\nas a selected subset A \u2713 E satis\ufb01es the constraint (A 2I V ), every\nsample is selected by at most k models.\nWith only the constraint A 2I v, different models can be assigned\ndramatically differently sized data subsets. In the extremely unbal-\nanced case, k models might get all the training data, while the other\nmodels get no data at all. This is obviously undesirable because k\nFigure 1: Data-Model Mar-\nmodels will learn from the same data (no diversity and no specialized\nriage as Bipartite Matching.\nand complementary expertise), while the others models learn nothing.\nRunning time also is not improved since training time is linear in the size of the largest assigned data\nset, which is all of the data in this case. We therefore introduce a second partition matroid constraint\nMU = (E,IU ), which limits to p the number of samples selected by each model. Speci\ufb01cally,\nIu = {A \u2713 E : |A \\ (u)|\uf8ff p,8u 2 U}. Eq. (1) then becomes:\n(2)\n\n\u2264'('=3)\n\nV\n\nE\n\nU\n\nmax\n{wi}m\n\ni=1\n\nmax\n\nA\u2713E,A2Iv\\Iu X(vj ,ui)2A\n\n( `(vj; wi)).\n\nInter-model & Intra-model Diversity\n\nThe interplay between the two constraints (i.e., Iv: k models per sample, and Iu: p samples per\nmodel) is important to our later design of a curriculum that leads to a diverse and complementary\nensemble. When mp < nk, Iu tends to saturate (i.e., |A \\ (u)| = p,8u 2 U) earlier than Iv.\nHence, each model generally has the opportunity to select the top-p easiest samples (i.e., those having\nthe smallest loss) for itself. We call this the \u201cmodel selecting sample\u201d (or early learning) phase, where\neach model quickly learns to perform well on a subset of data. On the other hand, when mp > nk, Iv\ntends to saturate earlier (i.e., |A\\ (v)| = k,8v 2 V ), and each sample generally has the opportunity\nto select the best-k models for itself. We call this the \u201csample selecting model\u201d (or later learning)\nphase, where models may develop complementary expertise so that together they can perform\naccurately over the entire data space. We give conditions on which phase dominates in Lemma 1.\n2.2\nTo encourage different multiple models to gain different pro\ufb01ciencies, the subsets of training data\nassigned to different models should be diverse. The two constraints introduced above are helpful\nto encourage diversity to a certain extent when p and k are small. For example, when k = 1 and\np \uf8ff n/m, no pairs of models share any training sample. Different training samples, however, can\nstill be similar and thus redundant, and in this case the above approach might not encourage diversity\nwhen p or k is large. Therefore, we incorporate during training an explicit inter-model diversity term\nFinter(A) ,Pi,j2[m],i G(A, W ) then\nA \u02c6A;\nend if\nfor i 2{ 1,\u00b7\u00b7\u00b7 , m} do\nr \u02c6H(wt\ni) @\ni + \u21e1\u21e3{w\u2327\nwt+1\ni wt\nend for\nt t + 1;\n\niPvj2V (A\\(ui)) `(vj; wt\ni )}\u23272[1,t]; \u2318t\u2318;\ni ,r \u02c6H(w\u2327\n\ni);\n\n@w t\n\ni=1, de\ufb01ne G(\u00b7, W ) by W ;\ni}m\n\n2The optimizer \u21e1(\u00b7; \u2318) can be any gradient descent methods, e.g., SGD, momentum methods, Nesterov\u2019s\naccelerated gradient [52], Adagrad [18], Adam [36], etc. Here the \ufb01rst parameter \u00b7 can include any historical\nsolutions and gradients, and \u2318 is a learning rate schedule (i.e., learning rate is \u2318t for iteration t).\n\n4\n\n\fIntuitively, we utilize the discrete optimization to select a better piece of \u02c6H(W ), and then apply the\ncontinuous optimization to \ufb01nd a better solution on that piece.\nDetails are given in Algorithm 1. For each iteration, we compute an approximate solution \u02c6A \u2713 E\nusing submodular maximization SUBMODULARMAX (line 6); in lines 7-9 we compare \u02c6A with the old\nA on G(\u00b7, W ) and choose the better one; lines 10-13 run an optimizer \u21e1(\u00b7; \u2318) to update each model\nwi according to its assigned data. Algorithm 1 always generates a non-decreasing (assuming \u21e1(\u00b7; \u2318)\ndoes the same using, say, a line search) sequence of objective values. With a damped learning rate,\nonly small adjustments get applied to W and G(\u00b7, W ). Thus, after a certain point the combinatorial\npart repeatedly selects the same A (and line 7 eventually is always false), so the algorithm then\nconverges as the continuous optimizer converges. 3\n3.2 Theoretical Perspectives\nAn interesting viewpoint of the max-max problem in Eq. (3) is its analogy to K-means problems [46].\nEq. (3) strictly generalizes the kmeans objective, by setting = = 0, k = 1, p to be the number\nof desired clusters, and the loss to be the distance metric used in K-means (e.g., L2 distance), and\nthe model to be a real valued vector of having the same dimension as x. Since K-means problem is\nNP-hard, our objective is also NP-hard.\nWe next analyze conditions for either of the constraints (Iv,Iu) introduced in Section 2.1 to saturate.\nIn the two extreme cases, we know that the \u201csample selecting model\u201d constraint Iv saturates when\nnk \u2327 mp (e.g., k = 1 and p = n), and the \u201cmodel selecting sample\u201d constraint Iu saturates when\nnk mp (e.g., k = m and p = 1). However, it is not clear what exactly happens between them.\nThe following Lemma shows the precise saturation conditions of the two constraints, with proof\ndetails in Section 5.1 of Appendix [71].\nLemma 1. If SUBMODULARMAX is greedy algorithm or its variant, the data assignment \u02c6A produced\nby lines 6-9 in Algorithm 1 ful\ufb01lls: 1) Iv saturates, i.e., | \u02c6A\\(v)| = k,8v 2 V , and | \u02c6A| = nk, if k <\nmp+p/n+(p1); 2) Iu saturates, i.e., | \u02c6A \\ (u)| = p,8u 2 U, and | \u02c6A| = mp, if k > mpp/n(p1);\n3) when mp+p/n+(p1) \uf8ff k \uf8ff mpp/n(p1), we have | \u02c6A| min{(k 1) + (m k + 1)p, (p \n1) + (n p + 1)k}.\nAs stated above, we can think the objective H(W ) as a piecewise function, where each piece is\nassociated with a solution to the discrete optimization problem. Since it is NP-hard to optimize the\ndiscrete problem, Algorithm 1 optimizes W on \u02c6H(W ), which is de\ufb01ned by the SUBMODULARMAX\nsolutions, rather than on H(W ). Algorithm 1 has the following properties.\nProposition 1. Algorithm 1: (1) generates a monotonically non-decreasing sequence of objective\nvalues G(A; W ) (assuming \u21e1(\u00b7; \u2318) does the same) (2) converges to a stationary point on \u02c6H(W ); and\n(3) for any loss `(u, w) that is -strongly convex w.r.t. w, if SUBMODULARMAX has approximation\nfactor \u21b5, it converges to a local optimum \u02c6W 2 argmaxW2K\n\u02c6H(W ) (i.e., \u02c6W is optimal in an local\narea K) such that for any local optimum W \u21e4loc 2K on the true objective H(W ), we have\n\u02c6H( \u02c6W ) \u21b5H(W \u21e4loc) +\n2. (4)\nThe proof is in Section 5.2 of Appendix [71]. The result in Eq. (4) implies that in any local area K,\nif \u02c6W is not close to W \u21e4loc (i.e., k \u02c6W W \u21e4lock2 is large), the algorithm can still achieve an objective\n\u02c6H( \u02c6W ) close to H(W \u21e4loc), which is a good approximate solution from the perspective of maximizing\nG(A, W ). Section 5.3 of Appendix [71] shows that the approximation factor is \u21b5 = 1/2+\uf8ffG for the\ngreedy algorithm, where \uf8ffG is the curvature of G(\u00b7, W ). When the weights and are small, \uf8ffG\ndecreases and G(\u00b7, W ) becomes more modular. Therefore, the approximation ratio \u21b5 increases and\nthe lower bound in Eq. (4) improves. For general non-convex losses and models (e.g., DNNs), Eq. (4)\ndegenerates to a weaker bound: \u02c6H( \u02c6W ) \u21b5H(W \u21e4loc).\n3.3 Ensemble Evolution: Curricula for Diverse Ensembles with Complementary Expertise\nFor a model ensemble to produce correct predictions, we require only that every sample be learnt\nby a few (small k) models. Optimizing Eq. (3) with small k from the beginning, however, might\nbe harmful as the models are randomly initialized, and using the loss of such early stage models\nfor the edge weights and small k could lead to arbitrary samples being associated and subsequently\n3Convergence is de\ufb01ned as the gradient r \u02c6H(W ) w.r.t. W being zero. In practice, we use kr \u02c6H(W )k \uf8ff \u270f\n\n\n2 \u00b7 min{(k 1) + (m k + 1)p, (p 1) + (n p + 1)k}\u00b7k \u02c6W W \u21e4lock2\n\nfor a small \u270f.\n\n5\n\n\flocked to models. We would, instead, rather have a larger k and more use of the diversity terms at the\nbeginning. To address this, we design an ensemble curriculum to guide the training process and to\ngradually approach our ultimate goal.\n\nj=1,{w0\n\ni }m\n\ni=1,\u21e1 (\u00b7; \u2318), \u00b5, k, p, T\n\ni\n\ni=1\n\ni}m\n 2 [0, 1], 2 [0, 1]\n\ni=1 SELECTLEARN(k, p, , , {wt1\ni}m\n\ni=1);\n{wt\n}m\n (1 \u00b5) \u00b7 , (1 \u00b5) \u00b7 ;\nk max{dk ke, 1}, p min{bp + pc, n};\n\nIn Section 2.1, we discussed two\nAlgorithm 2 Diverse Ensemble Evolution (DIVE2)\n(mp < nk and mp > nk) extreme\n1: Input: {(xj, yj)}n\ntraining regimes. In the \ufb01rst regime,\nthere are plenty of samples to go\n2: Output: {wt\naround but models may only choose a\n3: Initialize: k \uf8ff m, p 1 s.t. mp \uf8ff nk,\nlimited set of samples, so this encour-\nages different models to improve on\n4: for t 2{ 1,\u00b7\u00b7\u00b7 , T} do\nsamples they are already good at. In\n5:\nthe \ufb01rst regime, however, inter-model\n6:\ndiversity is important to encourage\n7:\nmodels to become suf\ufb01ciently differ-\n8: end for\nent from each other. Intra-diversity is\nalso important in the \ufb01rst regime, since it discourages models from being trained on entirely redundant\ndata. In the second regime, there are plenty of models to go around but samples may choose only\na limited number of models. Each model is then given a set of samples that it is particularly good\nat, and further training further specialization. Since all samples are assigned models, this leads to\ncomplementary pro\ufb01ciencies covering the data space.\nThese observations suggest we start at the \ufb01rst regime mp \uf8ff nk with small p and large k, and\ngradually switch to the second regime with mp nk by slowly increasing p and decreasing k. In\nearlier stages, we also should set the diversity weights and to be large, and then slowly reduce\nthem as we move towards the second regime. It is worth noting that besides intra-model diversity\nregularization, increasing p is also helpful to expand the expertise of each model since it encourages\neach model to select more diverse samples. Decreasing k also helps to encourage inter-model diversity\nsince it allows each sample to be shared by fewer models.\nIn later stages, the solution of Algorithm 1 becomes more exact. With and decreasing, according\nto Lemma 2, the curvature \uf8ffG of G(\u00b7, W ) approaches 0, the approximation factor \u21b5 = 1/2+\uf8ffG of\ngreedy algorithm increases, and the approximate objective \u02c6H(W ) \u21b5H(W ) becomes closer to the\ntrue objective H(W ). Moreover, during later stages when the \u201csample selecting model\u201d constraint\n(Iv) dominates and and are almost 0, the inner modular maximization is be exactly solved (\u21b5 = 1)\nand greedy algorithm degenerates to simple sorting.\nThe detailed diverse ensemble evolution (DivE2) procedure is shown in Algorithm 2. The curriculum\nis composed of T stages. Each stage uses SELECTLEARN (Algorithm 1) to (approximately) solve a\ncontinuous-combinatorial optimization in the form of Eq. (3) with pre-speci\ufb01ed values of (k, p, , )\nand initialization {wt1\ni=1 from the previous episode as a warm start (line 5). The procedure\n}m\nreduces and by a multiplicative factor (1 \u00b5) in line 6, linearly decreases k by k and additively\nincreases p by p, in Line 7. Both k and p are restricted to be integers and within the legal ranges, i.e.,\nk 2 [1, m] and p 2 [1, n]. The warm start initialization is similar in spirit to continuation schemes\nused in previous curriculum learning (CL) [6, 5, 4, 35, 2, 60, 70] and SPL [40, 64, 62, 65], to avoid\ngetting trapped in local minima and to stabilize optimization. As consecutive problems have the same\nform with similar parameters (k, p, , ), the solution to the previous problem might still evaluate\nwell on the next one. Hence, instead of running lines 5-14 in Algorithm 1 until full convergence (as\ninstructed by line 4), we run them for \uf8ff 10 iterations for reduced running time.\n4 Experiments\nWe apply three different ensemble training methods to train ensembles of neural networks with differ-\nent structures on four datasets, namely: (1) MobileNetV2 [56] on CIFAR10 [38]; (2) ResNet18 [29]\non CIFAR100 [38]; (3) CNNs with two convolutional layers4 on Fashion-MNIST (\u201cFashion\u201d in all\ntables) [69]; (4) and lastly CNNs with six convolutional layers on STL10 [12]5. The three training\nmethods include DivE2 and two widely used approaches as baselines, which are\n\u2022 Bagging(BAG)[7]: sample a new training set of the same size as the original one (with replacement)\n\nfor each model, and train it for several epochs on the sampled training set.\n\ni\n\n4A variant of LeNet5 with 64 kernels for each convolutional layer.\n5The network structure is from https://github.com/aaron-xichen/pytorch-playground.\n\n6\n\n\fon the whole training set.\n\n\u2022 RandINIT(RND): randomly initialize model weights of each model, and train it for several epochs\nDetails can be found in Table 3 of Appendix [71]. We everywhere \ufb01x the number of models at\nm = 10, and use `2 parameter regularization on w with weight 1 \u21e5 104. In DivE2\u2019s training\nphase, we start from k = 6, p = n/2m and linearly change to k = 1, p = 3n/m in T = 200 episodes.\nWe employ the \u201cfacility location\u201d submodular function [14, 45] for both the intra and inter-model\ndiversity, i.e., F (A) = Pv02V maxv2V (A) !v,v0 where !v,v0 represents the similarity between\nv, i.e., !v,v0 = expkz(v)z(v0)k2/22, where is the mean value of all the n(n1)/2 pairwise\n\ndistances. For every dataset, we train a neural networks on a small random subset of training data\n(e.g., hundreds of samples) for one epoch, and use the inputs to the last fully connected layer as\nfeatures z. These features are also used in the Top-k DCS-KNN approach (below) to compute the\npairwise `2 distances to \ufb01nd the K nearest neighbors.\n\nsample v and v0. We utilize a Gaussian kernel for similarity using neural net features z(v) for each\n\n4.1 Aggregation Methods using an Ensemble of Models\n\nCIFAR10\n\nCIFAR100\n\nFashion-MNIST\n\nSTL10\n\nFigure 2: Compare DivE2 with Bagging(upper row) and RandINIT(lower row) in terms of test accuracy (%) vs.\nnumber of training batches on CIFAR10, CIFAR100, Fashion-MNIST and STL10, with m = 10 and k = 3.\n\nFor ensemble model aggregation, when applying a trained ensemble of models to new samples, we\nmust determine (1) which models to use, and (2) how to aggregate their outputs. Here we mainly\ndiscuss the \ufb01rst point about different model selection methods, because the aggregation we employ\nis either an evenly or a weighted average of the selected model outputs. Static model selection\nmethods [72, 10, 53] choose a subset of models from the ensemble and apply it to all samples. By\ncontrast, dynamic classi\ufb01er selection (DCS) [11, 47, 73, 16] selects different subsets of models to be\naggregated for each sample. KNN based DCS [68, 37] is a widely used method that usually achieves\nbetter performance than other DCS and static methods. When training, DivE2 assigns different subsets\nof samples to different models, so for aggregation, we may bene\ufb01t more from using sample-speci\ufb01c\nmodel selection methods. Therefore, we focus on DCS-type methods, in particular, the following:\n\u2022 Top-k Oracle: average the outputs (e.g., logits before applying softmax) of the top-k models with\nthe smallest loss on the given sample. It requires knowing the true label, and thus is a cheating\nmethod that cannot be applied in practice. However, it shows a useful upper bound on the other\nmethods that select k models for aggregation.\n\n\u2022 All Average: evenly average the outputs of all m models.\n\u2022 Random-k Average: randomly select k models and average their outputs.\n\u2022 Top-k Con\ufb01dence: select the top-k models with the highest con\ufb01dence (i.e., highest probability of\n\u2022 Top-k DCS-KNN: apply an KNN based DCS method, i.e., \ufb01nd the K nearest neighbors of the\ngiven sample from the training data, select the top-k models assigned to the K nearest neighbors\nby Top-k Oracle, and average their outputs.\n\nthe predicted class) on the given sample, and average their outputs.\n\n\u2022 Top-k NN-LossPredict: train an L2-regression neural nets with m outputs to predict the per-sample\nlosses on the m models by using a training set composed of all training samples and their losses on\n\n7\n\n\fthe trained models. For aggregation, select the top-k models with the smallest predicted losses on\nthe given sample, and average their outputs.\n\naddition,\n\nCIFAR10\n\nCIFAR100\n\nDataset\nTotal time\nSUBMODULARMAX\n\nCIFAR10\n26790.75s\n1857.36s\n\nCIFAR100\n34658.27s\n2697.36s\n\nFashion\n2922.89s\n81.64s\n\nSTL10\n4065.81s\n378.84s\n\nTable 2: The highest test accuracy (%) achieved by different combinations of\nensemble training and aggregation methods on four datasets, with k = 3. DivE2\nusually requires less training time than others to achieve the highest accuracy. The\nbest non-cheating test accuracy (i.e., not Top-k Oracle) is highlighted below.\n\nWe compare the three training methods used with the aforementioned aggregation methods with\ndifferent k6. We summarize the highest test-set accuracy when k = 3 in Table 2, and show how the\ntest accuracy improves as\nTable 1: Total time (secs.) of DivE2 and time only on SUBMODULARMAX.\ntraining proceeds (i.e., as the\ntotal training batches on all\nmodels increases) in Fig. 2.\nIn Fig. 2, solid curves denote\nDivE2, while dashed curves\ndenote the three baseline training methods. Different colors refer to different aggregation methods,\nand gray curves represent single model performance (gray solid curves denote models trained by\nDivE2, while gray dashed curves denote models trained by other baselines). Similar results for k = 5\nand k = 7 can be found in Appendix [71]. In addition, we also tested DivE2 without the \u201cmodel\nselecting sample\u201d constraint and any diversity, which equals to [41, 26, 42] in multi-class case. It\nachieves a test accuracy of 90.11% (vs. 94.36% of DivE2) on CIFAR10 and 71.01% (vs. 78.89%\nof DivE2) on CIFAR100 when using Top-3 NN-LP for aggregation.\nTop-k Oracle (cheating)\nis always the best, and\nprovides an upper bound.\nIn\nDivE2\nusually has higher upper\nbound than others, and\nthus has more potential\nfor future improvement.\nSolid curves\n(DivE2)\nare usually higher than\ndashed curves\n(other\nbaselines)\nlater\nin\nstages, no matter which\naggregation method is\nused. Although diversity\nintroduces more dif\ufb01cult\nsamples and lead to\nslower convergence in\nearly stages,\nit helps\naccelerate convergence\nin later stages. Although\nthe test accuracy on\nsingle models achieved\nby DivE2 is usually lower than those obtained by other baselines, the test accuracy on the ensemble\nis better. This indicates that different models indeed develop different local expertise. Hence, each\nmodel performs well good only in a local region but poorly elsewhere. However, their expertise\nis complementary, so the overall performance of the ensemble outperforms other baselines. We\nvisualize the expertise of each model across different classes in Fig. 3 of Appendix [71] for\nFashion-MNIST as an example. Among all aggregation methods, Top-k NN-LossPredict and Top-k\nDCS-KNN show comparable or better performance than other aggregation methods, but require\nmuch less aggregation costs when k is small. As shown in Appendix [71], when changing k from\nminority (k = 3) to majority (k = 7), the test accuracy of these two aggregation methods usually\nimproves by a large margin. According to Table 1, DivE2 only requires a few extra computational\ntime for data assignment. The model training dominates the computations but is highly parallelizable\nsince the updates on different models are independent.\nAcknowledgments This material is based upon work supported by the National Science Foundation\nunder Grant No. IIS-1162606, the National Institutes of Health under award R01GM103544, and by\na Google, a Microsoft, and an Intel research award. This research is also supported by the CONIX\nResearch Center, one of six centers in JUMP, a Semiconductor Research Corporation (SRC) program\nsponsored by DARPA.\n\nTrain:Aggregation\nBAG:Top-k Oracle (Cheat)\nBAG:All Average\nBAG:Random-k Avg.\nBAG:Top-k Con\ufb01dence\nBAG:Top-k DCS-KNN\nBAG:Top-k NN-L.P.\nRND:Top-k Oracle (Cheat)\nRND:All Average\nRND:Random-k Avg.\nRND:Top-k Con\ufb01dence\nRND:Top-k DCS-KNN\nRND:Top-k NN-L.P.\nDivE2:Top-k Oracle (Cheat)\nDivE2:All Average\nDivE2:Random-k Avg.\nDivE2:Top-k Con\ufb01dence\nDivE2:Top-k DCS-KNN\nDivE2:Top-k NN-L.P.\n\nFashion\n95.60\n91.24\n91.00\n90.81\n91.39\n92.38\n95.71\n91.13\n90.77\n91.07\n92.01\n92.48\n96.40\n86.16\n82.75\n92.10\n92.10\n92.76\n\nSTL10\n89.13\n74.96\n74.03\n75.76\n74.07\n75.16\n89.54\n77.13\n76.75\n77.93\n77.23\n77.28\n90.18\n78.95\n78.59\n79.38\n79.23\n80.49\n\n97.85\n93.69\n93.05\n93.51\n92.86\n93.45\n97.80\n93.28\n93.11\n93.51\n93.18\n93.69\n98.01\n94.20\n93.26\n94.05\n93.81\n94.36\n\n88.02\n73.12\n72.86\n74.59\n73.06\n73.62\n87.01\n75.71\n75.56\n76.54\n75.72\n76.69\n90.12\n79.12\n77.69\n78.76\n77.61\n78.89\n\n6The k used in aggregation \ufb01xed, and is different from the k in training (which decreases from 6 to 1).\n\n8\n\n\fReferences\n\n[1] Wenruo Bai, Jeffrey Bilmes, and William S. Noble. Bipartite matching generalizations for\npeptide identi\ufb01cation in tandem mass spectrometry. In 7th ACM Conference on Bioinformatics,\nComputational Biology, and Health Informatics (ACM BCB), ACM SIGBio, Seattle, WA,\nOctober 2016. ACM, ACM SIGBio.\n\n[2] Sumit Basu and Janara Christensen. Teaching classi\ufb01cation boundaries to humans. In AAAI,\n\npages 109\u2013115, 2013.\n\n[3] Dhruv Batra, Payman Yadollahpour, Abner Guzman-Rivera, and Gregory Shakhnarovich.\n\nDiverse m-best solutions in markov random \ufb01elds. In ECCV, pages 1\u201316, 2012.\n\n[4] Yoshua Bengio. Evolving Culture Versus Local Minima, pages 109\u2013138. Springer Berlin\n\nHeidelberg, 2014.\n\n[5] Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and\nnew perspectives. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(8):1798\u2013\n1828, 2013.\n\n[6] Yoshua Bengio, J\u00e9r\u00f4me Louradour, Ronan Collobert, and Jason Weston. Curriculum learning.\n\nIn ICML, pages 41\u201348, 2009.\n\n[7] Leo Breiman. Bagging predictors. Machine Learning, 24(2):123\u2013140, 1996.\n\n[8] Leo Breiman. Random forests. Machine Learning, 45(1):5\u201332, 2001.\n\n[9] Cristian Bucilu\u02c7a, Rich Caruana, and Alexandru Niculescu-Mizil. Model compression. In\nProceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and\nData Mining, KDD \u201906, pages 535\u2013541, 2006.\n\n[10] Rich Caruana, Alexandru Niculescu-Mizil, Geoff Crew, and Alex Ksikes. Ensemble selection\nIn Proceedings of the Twenty-\ufb01rst International Conference on\n\nfrom libraries of models.\nMachine Learning, ICML \u201904, 2004.\n\n[11] Paulo R. Cavalin, Robert Sabourin, and Ching Y. Suen. Dynamic selection approaches for\n\nmultiple classi\ufb01er systems. Neural Computing and Applications, 22(3):673\u2013688, 2013.\n\n[12] Adam Coates, Honglak Lee, and Andrew Y. Ng. An analysis of single-layer networks in\n\nunsupervised feature learning. In AISTATS, pages 215\u2013223, 2011.\n\n[13] M. Conforti and G. Cornuejols. Submodular set functions, matroids and the greedy algorithm:\ntight worst-case bounds and some generalizations of the Rado-Edmonds theorem. Discrete\nApplied Mathematics, 7(3):251\u2013274, 1984.\n\n[14] G. Cornu\u00e9jols, M. Fisher, and G.L. Nemhauser. On the uncapacitated location problem. Annals\n\nof Discrete Mathematics, 1:163\u2013177, 1977.\n\n[15] Andrew Cotter, Mahdi Milani Fard, Seungil You, Maya Gupta, and Jeff Bilmes. Constrained\ninteracting submodular groupings. In International Conference on Machine Learning (ICML),\nStockholm, Sweden, July 2018.\n\n[16] Rafael M.O. Cruz, Robert Sabourin, and George D.C. Cavalcanti. Dynamic classi\ufb01er selection:\n\nRecent advances and perspectives. Information Fusion, 41:195\u2013216, 2018.\n\n[17] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei.\n\nHierarchical Image Database. In CVPR09, 2009.\n\nImageNet: A Large-Scale\n\n[18] John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning\n\nand stochastic optimization. Journal of Machine Learning Research, 12:2121\u20132159, 2011.\n\n[19] B. Efron. Bootstrap methods: Another look at the jackknife. The Annals of Statistics, 7(1):1\u201326,\n\n1979.\n\n9\n\n\f[20] Gamaleldin F. Elsayed, Shreya Shankar, Brian Cheung, Nicolas Papernot, Alex Kurakin, Ian J.\nGoodfellow, and Jascha Sohl-Dickstein. Adversarial examples that fool both human and\ncomputer vision. arXiv, 2018.\n\n[21] M. L. Fisher, G. L. Nemhauser, and L. A. Wolsey. An analysis of approximations for maximizing\n\nsubmodular set functions-II. Mathematical Programming Studies, 8, 1978.\n\n[22] Madalina Fiterau and Artur Dubrawski. Projection retrieval for classi\ufb01cation. In Advances in\n\nNeural Information Processing Systems 25, pages 3023\u20133031. 2012.\n\n[23] Yoav Freund and Robert E Schapire. A decision-theoretic generalization of on-line learning and\nan application to boosting. Journal of Computer and System Sciences, 55(1):119\u2013139, 1997.\n\n[24] Satoru Fujishige. Submodular functions and optimization. Annals of discrete mathematics.\n\nElsevier, 2005.\n\n[25] Jennifer Gillenwater, Alex Kulesza, and Ben Taskar. Near-optimal map inference for determi-\n\nnantal point processes. In NIPS, pages 2735\u20132743, 2012.\n\n[26] Abner Guzm\u00e1n-rivera, Dhruv Batra, and Pushmeet Kohli. Multiple choice learning: Learning\nto produce multiple structured outputs. In Advances in Neural Information Processing Systems\n25, pages 1799\u20131807. 2012.\n\n[27] Shizhong Han, Zibo Meng, AHMED-SHEHAB KHAN, and Yan Tong. Incremental boost-\ning convolutional neural network for facial action unit recognition. In Advances in Neural\nInformation Processing Systems (NIPS), pages 109\u2013117. 2016.\n\n[28] L. K. Hansen and P. Salamon. Neural network ensembles. IEEE Transactions on Pattern\n\nAnalysis and Machine Intelligence, 12(10):993\u20131001, 1990.\n\n[29] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image\nrecognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR),\npages 770\u2013778, 2016.\n\n[30] Geoffrey Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in a neural network.\n\nIn NIPS Deep Learning and Representation Learning Workshop, 2015.\n\n[31] Tin Kam Ho. Random decision forests. In Proceedings of 3rd International Conference on\n\nDocument Analysis and Recognition, volume 1, pages 278\u2013282, 1995.\n\n[32] Gao Huang, Yixuan Li, Geoff Pleiss, Zhuang Liu, John E. Hopcroft, and Kilian Q. Wein-\nberger. Snapshot ensembles: Train 1, get M for free. In International Conference on Learning\nRepresentations (ICLR), 2017.\n\n[33] Robert A. Jacobs, Michael I. Jordan, Steven J. Nowlan, and Geoffrey E. Hinton. Adaptive\n\nmixtures of local experts. Neural Computing, 3(1):79\u201387, 1991.\n\n[34] Michael I. Jordan and Robert A. Jacobs. Hierarchical mixtures of experts and the em algorithm.\n\nNeural Computing, 6(2):181\u2013214, 1994.\n\n[35] Faisal Khan, Xiaojin (Jerry) Zhu, and Bilge Mutlu. How do humans teach: On curriculum\n\nlearning and teaching dimension. In NIPS, pages 1449\u20131457, 2011.\n\n[36] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Interna-\n\ntional Conference on Learning Representations (ICLR), 2015.\n\n[37] Albert H. R. Ko, Robert Sabourin, and Alceu Souza Britto, Jr. From dynamic classi\ufb01er selection\n\nto dynamic ensemble selection. Pattern Recognition, 41(5):1718\u20131731, 2008.\n\n[38] Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images.\n\nTechnical report, University of Toronto, 2009.\n\n[39] Anders Krogh and Jesper Vedelsby. Neural network ensembles, cross validation, and active\nlearning. In Advances in Neural Information Processing Systems (NIPS), pages 231\u2013238. 1995.\n\n10\n\n\f[40] M. Pawan Kumar, Benjamin Packer, and Daphne Koller. Self-paced learning for latent variable\n\nmodels. In NIPS, pages 1189\u20131197, 2010.\n\n[41] Stefan Lee, Senthil Purushwalkam, Michael Cogswell, David J. Crandall, and Dhruv Batra.\nWhy m heads are better than one: Training a diverse ensemble of deep networks. arXiv,\nabs/1511.06314, 2015.\n\n[42] Stefan Lee, Senthil Purushwalkam Shiva Prakash, Michael Cogswell, Viresh Ranjan, David\nCrandall, and Dhruv Batra. Stochastic multiple choice learning for training diverse deep\nensembles. In Advances in Neural Information Processing Systems 29, pages 2119\u20132127. 2016.\n[43] Hui Lin and Jeff Bilmes. Word alignment via submodular maximization over matroids. In\nProceedings of the 49th Annual Meeting of the Association for Computational Linguistics:\nHuman Language Technologies: short papers-Volume 2, pages 170\u2013175. Association for\nComputational Linguistics, 2011.\n\n[44] Hui Lin and Jeff A. Bilmes. A class of submodular functions for document summarization. In\n\nACL, pages 510\u2013520, 2011.\n\n[45] Hui Lin, Jeff A. Bilmes, and Shasha Xie. Graph-based submodular selection for extractive\nsummarization. In Proc. IEEE Automatic Speech Recognition and Understanding (ASRU),\nMerano, Italy, December 2009.\n\n[46] S. Lloyd. Least squares quantization in pcm. IEEE Transactions on Information Theory (TIT),\n\n28(2):129\u2013137, 1982.\n\n[47] Christopher J. Merz. Dynamical Selection of Learning Algorithms, pages 281\u2013290. Springer\n\nNew York, 1996.\n\n[48] Michel Minoux. Accelerated greedy algorithms for maximizing submodular set functions.\nIn Optimization Techniques, volume 7 of Lecture Notes in Control and Information Sciences,\nchapter 27, pages 234\u2013243. Springer Berlin Heidelberg, 1978.\n\n[49] Baharan Mirzasoleiman, Ashwinkumar Badanidiyuru, Amin Karbasi, Jan Vondr\u00e1k, and Andreas\n\nKrause. Lazier than lazy greedy. In AAAI, pages 1812\u20131818, 2015.\n\n[50] Mohammad Moghimi, Mohammad Saberian, Jian Yang, Li-Jia Li, Nuno Vasconcelos, and\nSerge Belongie. Boosted convolutional neural networks. In British Machine Vision Conference\n(BMVC), 2016.\n\n[51] G. L. Nemhauser, L. A. Wolsey, and M. L. Fisher. An analysis of approximations for maximizing\n\nsubmodular set functions-I. Mathematical Programming, 14(1):265\u2013294, 1978.\n\n[52] Yurii Nesterov. Smooth minimization of non-smooth functions. Mathematical Programming,\n\n103(1):127\u2013152, 2005.\n\n[53] Ioannis Partalas, Grigorios Tsoumakas, and Ioannis Vlahavas. Focused ensemble selection: A\ndiversity-based method for greedy ensemble selection. In European Conference on Arti\ufb01cial\nIntelligence (ECML), pages 117\u2013121, 2008.\n\n[54] Adarsh Prasad, Stefanie Jegelka, and Dhruv Batra. Submodular meets structured: Finding\ndiverse subsets in exponentially-large structured item sets. In NIPS, pages 2645\u20132653, 2014.\n[55] Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100, 000+\n\nquestions for machine comprehension of text. In EMNLP, 2016.\n\n[56] Mark Sandler, Andrew G. Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen.\nInverted residuals and linear bottlenecks: Mobile networks for classi\ufb01cation, detection and\nsegmentation. arXiv, 2018.\n\n[57] Robert E. Schapire. The strength of weak learnability. Machine Learning, 5(2):197\u2013227, 1990.\n[58] Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton,\nand Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts\nlayer. In International Conference on Learning Representations (ICLR), 2017.\n\n11\n\n\f[59] Saurabh Singh, Derek Hoiem, and David Forsyth. Swapout: Learning an ensemble of deep\narchitectures. In Advances in Neural Information Processing Systems (NIPS), pages 28\u201336.\n2016.\n\n[60] Valentin I. Spitkovsky, Hiyan Alshawi, and Daniel Jurafsky. Baby Steps: How \u201cLess is\nMore\u201d in unsupervised dependency parsing. In NIPS 2009 Workshop on Grammar Induction,\nRepresentation of Language and Language Learning, 2009.\n\n[61] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov.\nDropout: A simple way to prevent neural networks from over\ufb01tting. Journal of Machine\nLearning Research (JMLR), 15:1929\u20131958, 2014.\n\n[62] James Steven Supancic III and Deva Ramanan. Self-paced learning for long-term tracking. In\n\nCVPR, pages 2379\u20132386, 2013.\n\n[63] C. Szegedy, Wei Liu, Yangqing Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke,\nand A. Rabinovich. Going deeper with convolutions. In Conference on Computer Vision and\nPattern Recognition (CVPR), volume 00, pages 1\u20139, 2015.\n\n[64] Kevin Tang, Vignesh Ramanathan, Li Fei-fei, and Daphne Koller. Shifting weights: Adapting\n\nobject detectors from image to video. In NIPS, pages 638\u2013646, 2012.\n\n[65] Ye Tang, Yu-Bin Yang, and Yang Gao. Self-paced dictionary learning for image classi\ufb01cation.\n\nIn MM, pages 833\u2013836, 2012.\n\n[66] F. Tram\u00e8r, A. Kurakin, N. Papernot, I. Goodfellow, D. Boneh, and P. McDaniel. Ensemble adver-\nsarial training: Attacks and defenses. In International Conference on Learning Representations\n(ICLR), 2018.\n\n[67] Andreas Veit, Michael J Wilber, and Serge Belongie. Residual networks behave like ensembles\nof relatively shallow networks. In Advances in Neural Information Processing Systems (NIPS),\npages 550\u2013558. 2016.\n\n[68] K. Woods, W. P. Kegelmeyer, and K. Bowyer. Combination of multiple classi\ufb01ers using\nlocal accuracy estimates. IEEE Transactions on Pattern Analysis and Machine Intelligence,\n19(4):405\u2013410, 1997.\n\n[69] Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for\n\nbenchmarking machine learning algorithms, 2017.\n\n[70] Tianyi Zhou and Jeff Bilmes. Minimax curriculum learning: Machine teaching with desirable\ndif\ufb01culties and scheduled diversity. In International Conference on Learning Representations\n(ICLR), 2018.\n\n[71] Tianyi Zhou, Shengjie Wang, and Jeff Bilmes. Supplementary material for diverse ensemble\n\nevolution. In NIPS, 2018.\n\n[72] Zhi-Hua Zhou, Jianxin Wu, and Wei Tang. Ensembling neural networks: Many could be better\n\nthan all. Arti\ufb01cial Intelligence, 137(1):239\u2013263, 2002.\n\n[73] Xingquan Zhu, Xindong Wu, and Ying Yang. Dynamic classi\ufb01er selection for effective\nmining from noisy data streams. In Data Mining, 2004. ICDM \u201904. Fourth IEEE International\nConference on, pages 305\u2013312, 2004.\n\n12\n\n\f", "award": [], "sourceid": 2854, "authors": [{"given_name": "Tianyi", "family_name": "Zhou", "institution": "University of Washington, Seattle"}, {"given_name": "Shengjie", "family_name": "Wang", "institution": "\"University of Washington, Seattle\""}, {"given_name": "Jeff", "family_name": "Bilmes", "institution": "University of Washington, Seattle"}]}