{"title": "XNAS: Neural Architecture Search with Expert Advice", "book": "Advances in Neural Information Processing Systems", "page_first": 1977, "page_last": 1987, "abstract": "This paper introduces a novel optimization method for differential neural architecture search, based on the theory of prediction with expert advice. Its optimization criterion is well fitted for an architecture-selection, i.e., it minimizes the regret incurred by a sub-optimal selection of operations. \nUnlike previous search relaxations, that require hard pruning of architectures, our method is designed to dynamically wipe out inferior architectures and enhance superior ones.\nIt achieves an optimal worst-case regret bound and suggests the use of multiple learning-rates, based on the amount of information carried by the backward gradients. \nExperiments show that our algorithm achieves a strong performance over several image classification datasets.\nSpecifically, it obtains an error rate of 1.6% for CIFAR-10, 23.9% for ImageNet under mobile settings, and achieves state-of-the-art results on three additional datasets.", "full_text": "XNAS: Neural Architecture Search\n\nwith Expert Advice\n\nNiv Nayman\u2217, Asaf Noy\u2217, Tal Ridnik\u2217, Itamar Friedman, Rong Jin, Lihi Zelnik-Manor\n\n{niv.nayman,asaf.noy,tal.ridnik,itamar.friedman,jinrong.jr,lihi.zelnik}\n\nMachine Intelligence Technology, Alibaba Group\n\n@alibaba-inc.com\n\nAbstract\n\nThis paper introduces a novel optimization method for differential neural architec-\nture search, based on the theory of prediction with expert advice. Its optimization\ncriterion is well \ufb01tted for an architecture-selection, i.e., it minimizes the regret in-\ncurred by a sub-optimal selection of operations. Unlike previous search relaxations,\nthat require hard pruning of architectures, our method is designed to dynamically\nwipe out inferior architectures and enhance superior ones. It achieves an optimal\nworst-case regret bound and suggests the use of multiple learning-rates, based on\nthe amount of information carried by the backward gradients. Experiments show\nthat our algorithm achieves a strong performance over several image classi\ufb01cation\ndatasets. Speci\ufb01cally, it obtains an error rate of 1.6% for CIFAR-10, 23.9% for\nImageNet under mobile settings, and achieves state-of-the-art results on three\nadditional datasets.\n\n1\n\nIntroduction\n\nIn recent years tremendous efforts have been put into a manual design of high performance neural\nnetworks [22, 16, 40, 39]. An emerging alternative approach is replacing the manual design with\nautomated Neural Architecture Search (NAS). NAS excels in \ufb01nding architectures which yield\nstate-of-the-art results. Earlier NAS works were based on reinforcement learning [55, 56], sequential\noptimization [24], and evolutionary algorithms [33], and required immense computational resources,\nsometimes demanding years of GPU compute time in order to output an architecture. More recent\nNAS methods reduce the search time signi\ufb01cantly, e.g. via weight-sharing [30] or by a continuous\nrelaxation of the space [25], making the search affordable and applicable to real problems.\nWhile current NAS methods provide encouraging results, they still suffer from several shortcomings.\nFor example, a large number of hyper-parameters that are not easy to tune, hard pruning decisions that\nare performed sub-optimally at once at the end of the search, and a weak theoretical understanding.\nThis cultivates skepticism and criticism of the utility of NAS in general. Some recent works even\nsuggest that current search methods are only slightly better than random search and further imply that\nsome selection methods are not well principled and are basically random [23, 35].\nTo provide more principled methods, we view NAS as an online selection task, and rely on Prediction\nwith Experts Advice (PEA) theory [4] for the selection. Our key contribution is the introduction of\nXNAS (eXperts Neural Architecture Search), an optimization method (section 2.2) that is well suited\nfor optimizing inner architecture weights over a differentiable architecture search space (section 2.1).\nWe propose a setup in which the experts represent inner neural operations and connections, whose\ndominance is speci\ufb01ed by architecture weights.\n\n\u2217These authors contributed equally.\n\n33rd Conference on Neural Information Processing Systems (NeurIPS 2019), Vancouver, Canada.\n\n\fOur proposed method addresses the mentioned shortcomings of current NAS methods. For the\nmitigation of the hard pruning, we leverage the Exponentiated-Gradient (EG) algorithm [21], which\nfavors sparse weight vectors to begin with, enhanced by a wipeout mechanism for dynamically\npruning inferior experts during the search process. Additionally, the algorithm requires less hyper-\nparameters to be tuned (section 3.2.2), and the theory behind it further provides guidance for the\nchoice of learning rates. Speci\ufb01cally, the algorithm avoids the decay of architecture weights [12],\nwhich is shown to promote selection of arbitrary architectures.\nAdditionally, XNAS features several desirable properties, such as achieving an optimal worst-case\nregret bound (section 3.1) and suggesting to assign different learning rates for different groups of\nexperts. Considering an appropriate reward term, the algorithm is more robust to the initialization of\nthe architecture weights and inherently enables the recovery of \u2019late bloomers\u2019, i.e., experts which\nmay become effective only after a warm-up period (section 3.2.1). The wipeout mechanism allows\nthe recovery of experts with a chance of being selected at the end of the process.\nWe compare XNAS to previous methods and demonstrate its properties and effectiveness over\nstatistical and deterministic setups, as well as over 7 public datasets (section 4). It achieves state-\nof-the-art performance over 3 datasets, and top-NAS over rest, with signi\ufb01cant improvements. For\nexample, XNAS reaches 1.60% error over CIFAR-10, more than 20% improvement over existing\nNAS methods.\n\n2 Proposed Approach\n\nTo lay out our approach we \ufb01rst reformulate the differentiable architecture search space of\nDARTS [25] in a way that enables direct optimization over the architecture weights. We then\npropose a novel optimizer that views NAS as an online selection task, and relies on PEA theory for\nthe selection.\n\n2.1 Neural Architecture Space\n\nWe start with a brief review of the PEA settings and then describe our view of the search space as\nseparable PEA sub-spaces. This enables us to leverage PEA theory for NAS.\nPEA Settings. PEA [4] refers to a sequential decision making framework, dealing with a decision\nmaker, i.e. a forecaster, whose goal is to predict an unknown outcome sequence {yt}T\nt=1 \u2208 Y while\nhaving access to a set of N experts\u2019 advises, i.e. predictions. Denote the experts\u2019 predictions at time\nt by ft,1, . . . , ft,N \u2208 D, where D is the decision space, which we assume to be a convex subset of\na vector space. Denote the forecaster\u2019s prediction {pt}T\nt=1 \u2208 D, and a non-negative loss function\n(cid:96) : D \u00d7 Y \u2212\u2192 R. At each time step t = 1, . . . , T , the forecaster observes ft,1, . . . , ft,N and predicts\npt. The forecaster and the experts suffer losses of (cid:96)t(pt) := (cid:96)(pt, yt) and (cid:96)t(ft,i) := (cid:96)(ft,i, yt)\nrespectively.\nThe Search Space Viewed as Separable PEA Sub-spaces. We view the search space suggested by\nDARTS [25] as multiple separable sub-spaces of experts, as illustrated in Figure 1, described next.\nAn architecture is built from replications of normal and reduction cells represented as a directed\nacyclic graph. Every node x(j) in this super-graph represents a feature map and each directed edge\n(j, k) is associated with a forecaster, that predicts a feature map p(j,k) := p(j,k)(x(j)) given the\ninput x(j). Intermediate nodes are computed based on all of their predecessors: x(k) = \u03a3j<kp(j,k).\nThe output of the cell is obtained by applying a reduction operation (e.g. concatenation) to the\nintermediate nodes. During the search stage, every forecaster combines N experts\u2019 feature map\npredictions {f (j,k)\n\ni=1 forming its own prediction,\n\n(x(j))}N\n\ni\n\np(j,k) =\n\nu(j,k)\ni\n\nf (j,k)\ni\n\n;\n\nu(j,k)\ni\n\n=\n\n;\n\nv(j,k)\ni\n\n\u2265 0\n\n(1)\n\ni=1 := {f (j,k)\n}N\nN(cid:88)\n\ni\n\ni=1\n\ni(cid:80)N\n\nv(j,k)\nl=1 v(j,k)\n\nl\n\nFrom now on, we will ignore the superscript indices (j, k) for brevity. Each expert represents a\nneural operation, e.g. convolution or max pooling, associated with network weights wi, that receives\nan input xt at time t, and outputs a feature map prediction. Thus a time index is attached, as each\nprediction ft,i := fi(xt) is associated with updated weights wt,i.\n\n2\n\n\fOur architecture search approach is composed of two stages. In the search stage, the weights wt,i and\nvt,i are alternately optimized as described in section 2.2; then, in the discretization stage, a discrete\nchild architecture is obtained as explained next.\nThe Discretization Stage. Once the architecture weights are optimized, the \ufb01nal discrete neural\narchitecture is obtained by performing the following discretization stage, adopted from [25]: Firstly,\nthe strongest two predecessor edges are retained for each intermediate node. The strength of an edge\nis de\ufb01ned as maxi u(k,j)\n. Lastly, every forecaster is replaced by the corresponding strongest expert.\n\ni\n\n2.2 XNAS: eXperts Neural Architecture Search\n\nThe differential space, described in section 2.1, enables direct optimization over the architecture\nweights via gradient-descent based techniques. Previous methods adopted generic optimizers com-\nmonly used for training the network weights. For example [25, 49, 6, 3] used adam [20], and [29]\nused SGD with momentum. While those optimizers excel in joint minimization of neural network\nlosses when applied to network weights, NAS is a essentially a selection task, aiming to select a\nsubset of experts out of a superset. The experts weights form a convex combination, as they compete\nover a forecaster\u2019s attention.\nWe argue that a generic alternate optimization of network weights and architecture weights, as\nsuggested in previous works, e.g. [17, 25], is not suitable for the unique structure of the architecture\nspace. Hence, we design a tailor-made optimizer for this task, inspired by PEA theory. In order to\nevaluate experts\u2019 performance, a loss is to be associated with each expert. However, an explicit loss\nis not assigned to each expert, as opposed to a back-propagated loss gradient. Therefore, we base our\nalgorithm on a version of the Exponentiated-Gradient (EG) algorithm adapted for the NAS space.\nEG algorithms favor sparse weight vectors [21], thus \ufb01t well to online selection problems.\nWe introduce the XNAS (eXperts Neural Architecture Search) algorithm, outlined in Algorithm 1 for\na single forecaster. XNAS alternates between optimizing the network weights w and the architecture\nweights v in a designated manner. After updating w (line 4) by descending the train loss (cid:96)train,\neach forecaster makes predictions over the validation set, based on the associated mixture of experts\n(line 5), as they propagate forward through the super-graph, incurring a validation loss (cid:96)val. Then v\nare updated according to the EG rule (lines 8-10) with respect to the validation loss, whose gradients\nare backward propagated to every forecaster through the super-graph (line 6). Next, the optimizer\nwipes-out weak experts (lines 12-13), effectively assigning their weights to the remaining ones. The\nexponential update terms, i.e. rewards, are determined by the projection of the loss gradient on the\nexperts\u2019 predictions: Rt,i = \u2212\u2207pt(cid:96)val(pt) \u00b7 ft,i. In section 3.1 we relate to the equivalence of this\nreward to the one associated with policy gradient search [45] applied to NAS.\n\nAlgorithm 1 XNAS for a single forecaster\n1: Input: The learning rate \u03b7, Reward bound L ,\ni=1 \u2200t = 1, . . . , T\nExperts predictions {ft,i}N\n2: Init: I0 = {1, . . . , N}, v0,i \u2190 1, \u2200i \u2208 I0\n3: for rounds t = 1, . . . , T do\nUpdate \u03c9 by descending \u2207\u03c9(cid:96)train(\u03c9, v)\n4:\npt \u2190\n#Predict\ni\u2208It\u22121\n{loss gradient revealed: \u2207pt(cid:96)val(pt)}\nfor i \u2208 It\u22121 do\n\nvt\u22121,i\u00b7ft\u22121,i\n\ni\u2208It\u22121\n\n(cid:80)\n\n(cid:80)\n\nvt\u22121,i\n\n5:\n\n6:\n7:\n8:\n9:\n10:\n11:\n12:\n\n#Rewards\n\nRt,i = \u2212\u2207pt(cid:96)val(pt) \u00b7 ft,i\nRt,i \u2190 min{max{Rt,i,\u2212L} ,L}\nvt,i \u2190 vt\u22121,i \u00b7 exp{\u03b7Rt,i}\n\nend for\n\u03b8t \u2190 max\ni\u2208It\u22121\nIt \u2190 It\u22121 \\ {i | vt,i < \u03b8t}\n\n#EG step\n{vt,i} \u00b7 exp{\u22122\u03b7L(T \u2212 t)}\n#Wipeout\n\n13:\n14: end for\n\nFigure 1: A visual illustration of a single forcaster,\nequipped with XNAS forward and backward prop-\nagation. Forward and backward passes are pre-\nsented as red and blue arrows respectively.\n\n3\n\n\fThe purpose of the wipeout step is threefold. First, the removal of weak experts consistently smooths\nthe process towards selecting a \ufb01nal architecture at the descretization stage described in section\n2.1. Thus it mitigates the harsh \ufb01nal pruning of previous methods, which results in a relaxation\nbias addressed in [50, 29]. Second, it dynamically reduces the number of network weights, thus\nsimplifying the optimization problem, avoiding over-\ufb01tting and allowing it to converge to a better\nsolution. Last, it speeds-up the architecture search, as the number of graph computations decreases\nwith the removal of experts.\n\n3 Analysis and Discussion\n\n3.1 Theoretical Analysis\n\nIn this section we analyse the performance of the proposed algorithm. For this purpose, we introduce\nthe regret as a performance measure for NAS algorithms. Showing that the wipeout mechanism\ncannot eliminate the best expert (lemma 1), we provide theoretical guarantees for XNAS with respect\nto that measure (theorem 1). Proofs appear in Section 7 of the supplementary material for brevity.\nRelaying on the theoretical analysis, we extract practical instructions with regard to the choice of\nmultiple learning rates. Finally we brie\ufb02y discuss the equivalence of our reward to the one considered\nby the policy gradient approach applied to NAS.\nThe Regret as a Performance Measure. Denote the regret, the cumulative losses of the forecaster\nand of the ith expert at time t by,\n\nt(cid:88)\n\nt(cid:88)\n\nRegrett = Lt \u2212 min\n\ni=1,...N\n\nLt,i\n\n; Lt =\n\n(cid:96)s(ps)\n\n; Lt,i =\n\n(cid:96)s(fs,i)\n\n(2)\n\ns=1\n\ns=1\n\nrespectively. The regret measures how much the forecaster regrets not following the advice of the\nbest expert in hindsight. This criterion suits our setup as we optimize a mixture of experts and select\nthe best one by the end of the process.\nIn classical learning theory, statistical properties of the underlying process may be estimated on the\nbasis of stationarity assumptions over the sequence of past observations. Thus effective prediction\nrules can be derived from these estimates [4]. However, NAS methods that alternately learn the\narchitecture weights and train the network weights are highly non-stationary. In PEA theory, no\nstatistical assumptions are made, as \u201csimplicity is a merit\u201d [14], and worst-case bounds are derived\nfor the forecaster\u2019s performance. We obtain such bounds for the wipeout mechanism and the regret.\nA Safe Wipeout. In XNAS (line 12), by the choice of the wipeout thresholds, experts with no chance\nof taking the lead by the end of the search are wiped out along the process. In a worse-case setup, a\nsingle incorrect wipeout might result in a large regret, i.e., linear in the number of steps T , due to a\nloss gap at each consecutive step. The following lemma assures that this cannot happen,\nLemma 1. In XNAS, the optimal expert in hindsight cannot be wiped out.\n\n(cid:80)\n\nand \u03b3T :=(cid:81)T\n\nvt,i\n\ni\u2208It\u22121\\It\nvt,i\n\ni\u2208It\n\n(cid:80)\n\nThe wipeout effectively transfers the attention to leading experts. De\ufb01ne the wipeout factor and the\naggregated wipeout factor as \u0393t := 1 +\nLemma 2. The aggregated wipeout factor satis\ufb01es 1 \u2264 \u03b3T < N.\nEquipped with both lemmas, we show that the wipeout may improve the EG regret bound for certain\nreward sequences.\nRegret Bounds. Our main theorem guarantees an upper bound for the regret,\nTheorem 1 (XNAS Regret Bound). The regret of the XNAS algorithm 1, with N experts and learning\nrate \u03b7, incurring a sequence of T non-negative convex losses of L-bounded rewards, satis\ufb01es,\n\nt=1 \u0393t, respectively.\n\nRegretT \u2264 \u03b7TL2\n\n2\n\n+\n\n1\n\u03b7\n\nln N \u2212 1\n\u03b7\n\nln \u03b3T\n\n(3)\n\nAs an input parameter of XNAS, the learning rate \u03b7 cannot be determined based on the value of \u03b3T ,\nsince the later depends on the data sequence. Choosing the minimizer \u03b7\u2217 of the \ufb01rst two terms of (3)\n\n4\n\n\ffully known in advance, yields the following tight upper bound,\n\n(cid:114)\n\n\u03b7\u2217 =\n\n2 ln N\nTL2\n\n\u221a\n; RegretT \u2264 L\n\n2T ln N\n\n(cid:18)\n\n1 \u2212 1\n2\n\nln \u03b3T\nln N\n\n(cid:19)\n\n(4)\n\nT ln N ) [13].\n\n\u2126(L\u221a\nbound of gradient-descent is of O(L\u221a\n\nThe regret upper bound of XNAS is tight, as the lower bound can be shown to be of\nIn addition, the wipeout related term reduces the regret in an amount\nwhich depends on the data sequences through \u03b3T , as it effectively contributes the attention of weak\nexperts to the leading ones. For comparison, under the same assumptions, the worst-case regret\nT N ) [14], while the one of Adam is linear in T [34]. An\nillustration of the relationship between the regret and the rate of correct expert selection appears in\nsection 8.3 of the supplementary material, where XNAS is shown to achieve a better regret compared\nto a generic optimizer.\n\nMultiple Learning Rates. Equation 4 connects the optimal theoretical learning rate \u03b7\u2217 with\nthe number of steps T , which is also the number of gradient feedbacks received by the experts.\nSince forecasters weights are being replicated among different cells, the number of feedbacks\nis different for normal and reduction cells (section 2.1). Explicitly, Tc = d \u00d7 E \u00d7 rc, where\nTc, d,E, rc are the effective horizon T , the validation set size, the number of epochs and the\nnumber of replications for cell type c respectively. We adopt the usage of multiple learning rates \u03b7\u2217\nc\nin our experiments as upper bounds on the learning rates for minimizing the upper bound of the regret.\n\nThe Connection to Policy Gradient. We conclude this section by pointing out an interesting\nconnection between policy gradient in NAS [55] and PEA theory. We refer to the PEA based reward\nterm in line 8 of algorithm 1. This reward has been shown by [50] to be the same effective reward\nof a policy gradient method applied to the common NAS optimization criterion [55, 56, 30]. More\nprecisely, consider the case where instead of mixing the experts\u2019 predictions, the forecaster is to\nsample a single expert at each step with probability ut,i, speci\ufb01ed in (1). Then the effective reward\nassociated with the policy gradient will be exactly the derived PEA reward, Rt,i = \u2212\u2207pt(cid:96)t(pt) \u00b7 ft,i.\nXNAS optimizes with respect to the same reward, while avoiding the sampling inef\ufb01ciency associated\nwith policy gradient methods.\n\n3.2 Key Properties and Discussion\n\nIn this section we discuss some of the key properties of XNAS. For each of this properties we provide\nsupporting derivations, illustrations and demonstrations appearing in section 8 of the supplementary\nmaterial for brevity.\n\n3.2.1 The Recovery of Late Bloomers and Robustness to Initialization\n\nIn this section we point out a key difference between our proposed update rule and the one used\nin previous works. We refer to Gradient Descent (GD) with softmax as updating the parameters\n\u03b1t,i = ln vt,i, by descending \u2207\u03b1t,i(cid:96)t(pt) respectively. Variants of GD with softmax, as used in [25]\nto optimize the architecture weights, suppress operations that are weak at the initial iterations, making\nit more dif\ufb01cult for them to \u201cbloom\u201d and increase their weights. This could be problematic, e.g. in the\ntwo following cases. First, consider the best expert starting with a poor weight which gradually rises.\nThis could be the case when an expert representing a parameterized operation (e.g. a convolutional\nlayer) competes with an unparameterized one (e.g. a pooling layer), as the \ufb01rst requires some period\nfor training, as stated by [29, 17]. Second, consider a noisy setup, where the best expert in hindsight\ncould receive some hard penalties before other inferior experts do. In NAS we deal with stochastic\nsettings associated with the training data.\nWe inspect the update term of GD with softmax,\n\nvt,i = exp{\u03b1t,i} = exp{\u03b1t\u22121,i \u2212 \u03b7\u2202\u03b1t\u22121,i(cid:96)t(pt)} = vt\u22121,i \u00b7 exp{\u2212\u03b7\u2202\u03b1t\u22121,i (cid:96)t(pt)}\n\n(5)\n\nHence, the effective reward in this case is,\n\n\u02dcRt,i := \u2212\u2202\u03b1t\u22121,i (cid:96)t(pt) = \u2212\u2207pt(cid:96)t(pt) \u00b7 ut\u22121,i (ft,i \u2212 pt)\n\n(6)\nSee derivations in section 8.4. The linear dependence on the expert\u2019s weight ut\u22121,i in (6) implies that\nGD with softmax makes it harder for an expert whose weight is weak at some point to recover and\nbecome dominant later on, as the associated rewards are attenuated by the weak expert\u2019s weight.\n\n5\n\n\fXNAS mitigates this undesirable behavior. Since for XNAS the update term (8) depends on the\narchitecture weights only indirectly, i.e. through the prediction, the recovery of late bloomers is\nnot discouraged, as demonstrated in section 8.1 of the supplementary material. From the very same\nreasons, XNAS is more robust to the initialization scheme compared to GD with softmax and its\nvariants, as demonstrated in section 8.2 of the supplementary material. These advantages make XNAS\nmore suitable for the NAS setup.\nNote that while the XNAS enables the recovery of experts with badly initialized weights or with\ndelayed rewards, the wipeout mechanism prevents inferior operations that start blooming too late\nfrom interfering, by eliminating experts with no chance of leading at the end.\nWipeout Factor. As mentioned in section 2.2, the wipeout mechanism contributes to both opti-\nmization process and search duration. A further reduction in duration can be achieved when the\nwipe-out threshold in line 12 of Algorithm 1 is relaxed with a parameter 0 < \u03b6 \u2264 1, being replaced\nby \u03b8t \u2190 maxi\u2208It\u22121{vt,i}\u00b7 exp{\u22122\u03b7L(T \u2212 t) \u00b7 \u03b6}. This will lead to a faster convergence to a single\narchitecture, with the price of a violation of the theoretical regret. As worst-case bounds tend to be\nover pessimistic, optimizing over \u03b6 could lead to improved results. We leave that for future work.\n\nN =7.5e-4 and \u03b7\u2217\n\nR =1.3e-3.\n\n3.2.2 Fewer Hyper Parameters\nThe view of the differentiable NAS problem as an optimization problem solved by variants of GD,\ne.g. Adam, introduces some common techniques for such schemes along with their corresponding\nhyper-parameters. Tuning these complicates the search process - the fewer hyper-parameters the\nbetter. We next discuss how XNAS simpli\ufb01es and reduces the number of hyper-parameters.\nTheoretically Derived Learning Rates. The determination of the learning rate has a signi\ufb01cant\nimpact on the convergence of optimization algorithms. Various scheduling schemes come up, e.g.\n[26, 38], as the later additionally suggests a way for obtaining an empirical upper bound on the\nlearning rate. In section 3.1, multiple learning rates \u03b7\u2217\nc are suggested for minimizing the regret bound\n(4), as c \u2208 {N, R} represents normal and reduction cells respectively. For example, for CIFAR10\nwith 50%:50% train-validation split, 50 search epochs, gradient clipping of 1, 6 normal cells and 2\nreduction cells both of 8 experts for each forecaster, (4) yields \u03b7\u2217\nRemark 1. Note that the proposed learning rates minimize an upper bound of the regret (4) in the\ncase of no wipeout, i.e. the worst case, as the extent of the wipeout cannot be known in advance.\nHence the proposed learning rate provides an upper bound on the optimal learning rates and can be\nfurther \ufb01ne-tuned.\nNo Weight Decay. Another common tech-\nnique involving hyper-parameters is weight de-\ncay, which has no place in the theory behind\nXNAS. We claim that the obviation of weight\ndecay by XNAS makes sense. Regularization\ntechniques, such as weight decay, reduce over-\n\ufb01tting of over-parametrized models when ap-\nplied to these parameters [12]. No such effect\nis incurred when applying weight decay on the\narchitecture parameters as they do not play the\nsame role as the trained network parameters w.\nInstead, weight decay encourages uniform dense\nsolutions, as demonstrated in Figure 2, where\nthe mean normalized entropy increases with the\nweight decay coef\ufb01cient. The calculation of the\nmean normalized entropy is detailed in section\n8.5 of the supplementary material. This observation could be associated with the suggestion of recent\nworks [23, 35] that current search methods are only slightly better than random search. The density\nof results in a harder degradation in performance once discretization stage occurs (section 2.1), hence\nsparse solutions are much preferred over dense ones.\nNo Momentum. The theory behind XNAS obviates momentum [31] and ADAM\u2019s exponentially\ndecay rates [20]. Since momentum requires more state variables and more computations, the resulting\nXNAS optimizer turns out to be simpler, faster and with a smaller memory footprint, compared to\ncommonly used optimizers for NAS, e.g. ADAM [25, 49, 6, 3] and SGD with momentum [29].\n\nFigure 2: Mean normalized entropy vs weight de-\ncay. The red dot refers to DARTS\u2019 settings.\n\n6\n\n\f4 Experiments and Results\n\nIn this section we will test XNAS on common image classi\ufb01cation benchmarks, and show its\neffectiveness compared to the other state-of-the-art models.\nWe used the CIFAR-10 dataset for the main search and evaluation phase. In addition, using the cell\nfound on CIFAR-10 we did transferability experiments on the well-known benchmarks ImageNet,\nCIFAR-100, SVHN, Fashion-MNIST, Freiburg and CINIC10.\n4.1 Architecture Search on CIFAR-10\nUsing XNAS, we searched on CIFAR-10 in a small parent network for convolutional cells. Then we\nbuilt a larger network by stacking the learned cells, trained it on CIFAR-10 and compared the results\nagainst other NAS methods.\nWe created the parent network by stacking 8 cells with 4 ordered nodes, each of which connected\nvia forecasters to all previous nodes in the cell and also to the two previous cells outputs. Each\nforecaster contains seven operations: 3x3 and 5x5 separable and dilated separable convolutions, 3x3\nmax-pooling, 3x3 average-pooling and an identity. A cells output is a concatenation of the outputs of\nthe four cells nodes.\nThe search phase lasts up to 50 epochs. We use the \ufb01rst-order approximation [25], relating to v and\n\u03c9 as independent parameters which can be optimized separately. The train set is divided into two\nparts of equal sizes: one is used for training the operations weights \u03c9 and the other for training the\narchitecture weights v, both with respect to the cross entropy loss. With a batch size of 96, one epoch\ntakes 8.5 minutes in average on a single GPU2 , summing up to 7 hours in total for a single search.\nMost of the speedup over related differentiable NAS methods is achieved by the quick EG updates\nand the rest by the wipeout. Figure 9 from the supplementary material shows a searched normal and\nreduction cells that will be used in the following evaluations.\n4.2 CIFAR-10 Evaluation Results\nWe built the evaluation network by stacking 20 cells:\n18 normal cells and 2 reduction cells. The reduction\ncells are placed after 1/3 and 2/3 of the network. Af-\nter each reduction cell we double the amount of chan-\nnels in the network. We trained the network for 1500\nepochs using a batch size of 96 and SGD optimizer\nwith nesterov-momentum. Our learning rate regime\nwas composed of 5 cycles of power cosine anneal-\ning learning rate [17], with amplitude decay factor of\n0.5 per cycle. For regularization we used cutout [9],\nscheduled drop-path [22], auxiliary towers [39], label\nsmoothing [40] AutoAugment [7] and weight decay.\nTo understand the effect of the network size on the\n\ufb01nal accuracy, we chose to test three architecture con-\n\ufb01gurations, XNAS-Small, XNAS-Medium and XNAS-\nLarge, with 36, 44 and 50 initial network channels\nrespectively. Table 1 shows the performance of XNAS\narchitectures compared to other state-of-the-art NAS\nmethods. XNAS\u2019 smallest network variant, XNAS-\nSmall, outperforms previous NAS methods by a large\nmargin, with a minimal error rate of 1.81% measured\nacross 5 seeds with error rate mean and standard devia-\ntion of 1.91% \u00b1 0.07%. Training the same model with\nthe settings of [25] yields 2.48% error rate. Our largest\nnetwork variant, XNAS-Large, is the second highest\nreported score on CIFAR-10 (without any additional\npre-train data), while having 4 times less parameters\nthan the top one [7]3. In addition, XNAS is among the\nfastest NAS methods\n\n1800\n3150\n0.3\n200\n10\n0.2\n0.8\n0.3\n0.3\n0.3\nTable 1: Classi\ufb01cation errors of XNAS com-\npared to state-of-the-art methods on CIFAR-\n10. The Search cost is measured in GPU days.\nTest error refers to top-1 test error (%). XNAS\nSmall, Medium and Large refer to network\narchitectures with XNAS cells and 36, 44 and\n50 initial channels respectively.\n\nCIFAR-10\nArchitecture\nAutoAugment [7]\nNAONet-WS [27]\nPNAS [24]\nAmoeba-A [33]\nDSO-NAS [53]\nDARTS(1nd) [25]\nENAS [30]\nSNAS [50]\nDARTS(2nd) [25]\nNASNet-A [56]\nAmoeba-B [33]\nPDARTS [6]\nNAONet [27]\nProxylessNAS [2]\nASAP [29]\nSharpDarts [17]\nXNAS-Small\nXNAS-Medium\nXNAS-Large\n\nSearch\ncost\n20\n0.3\n150\n3150\n\n1\n0.4\n0.5\n1.5\n1\n\nTest\nerror\n1.48\n3.53\n3.41\n3.34\n2.95\n2.94\n2.89\n2.85\n2.67\n2.65\n2.55\n2.50\n2.11\n2.08\n1.68\n1.93\n1.81\n1.73\n1.60\n\nParams\n(M)\n26\n2.5\n3.2\n3.2\n3\n2.9\n4.6\n2.8\n3.4\n3.3\n2.8\n3.4\n128\n5.7\n6.0\n3.6\n3.7\n5.6\n7.2\n\n2Experiments were performed using a NVIDIA GTX 1080Ti GPU.\n\n7\n\n\f4.3 Transferability Evaluation\nUsing the cell found by XNAS search on CIFAR-\n10, we preformed transferability tests on 6 popular\nclassi\ufb01cation benchmarks: ImageNet, CIFAR-100,\nFashion-MNIST, SVHN, Freiburg and CINIC10.\nImageNet Results. Our ImageNet network was\ncomposed of 14 stacked XNAS cells, with two ini-\ntial stem cells for downscaling. We used 46 initial\nchannels so the total number of network FLOPs\nis below 600[M]3, similar to other ImageNet ar-\nchitectures with small computation regime [48].\nWe trained the network for 250 epochs with one\ncycle of power cosine learning rate and a nesterov-\nmomentum optimizer. The results are presented\nin Table 2. We can see from Table 2 that XNAS\ntransferability results on ImageNet are highly com-\npetitive, outperforming all previous NAS cells.\n\nImageNet\nArchitecture\nSNAS [50]\nASAP [29]\nDARTS [25]\nNASNet-A [56]\nPNAS [24]\nAmoeba-A [33]\nRandWire [48]\nSharpDarts [17]\nAmoeba-C [33]\nXNAS\n\nTest\nerror\n27.3\n24.4\n26.7\n26.0\n25.8\n25.5\n25.3\n25.1\n24.3\n23.9\n\nParams\n(M)\n4.3\n5.1\n4.9\n5.3\n5.1\n5.1\n5.6\n4.9\n6.4\n5.2\n\nSearch\ncost\n1.5\n0.2\n1\n\n1800\n150\n3150\n\n0\n0.8\n3150\n0.3\n\nTable 2: Transferability classi\ufb01cation error\nof XNAS, compared to top NAS cells, on\nImageNet. Test error refers to top-1 test error\n(%). Search cost is measured in GPU days.\n\nAdditional Results. We further tested XNAS transferability abilities on 5 smaller datasets: CIFAR-\n100 [42], Fashion-MNIST [47], SVHN [28], Freiburg [19] and CINIC10 [8]. We chose to use the\nXNAS-Small architecture, with similar training scheme to the one described in section 4.2. Table 3\nshows the performance of our model compared to NAS methods. We can see that XNAS cell\nexcels on the datasets tested. On CIFAR-100 it surpasses the next top cell by 1%, achieving the\nsecond highest reported score on CIFAR-100 (without additional pre-train data), second only to [7].\nOn Fashion-MNIST, Freiburg and CINIC10, to the best of our knowledge XNAS achieves a new\nstate-of-the-art accuracy.\n\nDatasets\nArchitecture\nKnown SotA\nPDARTS [6]\nNAONet-1 [27]\nNAONet-2 [27]\nPDARTS-L [6]\nSNAS\u2020 [50]\nPNAS\u2020 [24]\nAmoeba-A\u2020 [33]\nNASNet\u2020 [56]\nDARTS\u2020 [25]\nASAP\u2020 [29]\nXNAS-Small\n\n15.9\n15.7\n14.7\n14.6\n16.5\n15.9\n15.9\n15.8\n15.7\n15.6\n13.6\n\nCIFAR100\n\nFMNIST\n\nError\n\n10.7 [7]\n\nError\n\n3.65 [54]\n\nSVHN\nError\n1.02 [7]\n\n-\n-\n-\n-\n\n1.98\n1.83\n1.93\n1.96\n1.95\n1.81\n1.72\n\nFreiburg\n\nError\n\n10.7 [29]\n\n-\n-\n-\n-\n\nCINIC10\n\nError\n\n6.83 [29]\n\n-\n-\n-\n-\n\n14.7\n12.3\n11.8\n13.4\n10.8\n10.7\n6.3\n\n7.13\n7.03\n7.18\n6.93\n6.88\n6.83\n6.0\n\nParams\n\n(M)\n26 [7]\n3.6\n10.8\n128\n11\n2.8\n3.4\n3.2\n3.3\n3.4\n2.5\n3.7\n\nSearch\ncost\n\n-\n0.3\n200\n200\n0.3\n1.5\n150\n3150\n1800\n\n1\n0.2\n0.3\n\n-\n-\n-\n-\n\n3.72\n3.72\n3.8\n3.71\n3.68\n3.73\n3.64\n\nTable 3: Classi\ufb01cation errors of XNAS, compared to state-of-the-art NAS methods, on several\ndatasets. Error refers to top-1 test error (%). The Search cost is measured in GPU days. Results\nmarked with \u2020 are taken from [29], which tested and compared different cells on various datasets.\n\n5 Related Work\n\nMentions of Experts in deep learning [32, 52, 11, 1] literature go decades back [18, 5], typically\ncombining models as separable experts sub-models. A different concept of using multiple mixtures of\nexperts as inner parts of a deep model, where each mixture has its own gating network, is presented in\n[10]. Following works build upon this idea and include a gating mechanism per mixture [51, 41], and\nsome further suggest sparsity regularization over experts via the gating mechanism [37, 44]. These\n\n3XNAS evaluation results can be reproduced using the code: https://github.com/NivNayman/XNAS\n\n8\n\n\fgating mechanisms can be seen as a dynamic routing, which activates a single or a group of experts in\nthe network on a per-example basis. Inspired by these works, our methods leverage PEA principled\nmethods for automatically designing neural network inner components.\nFurthermore, optimizers based on PEA theory may be useful for the neural architecture search\nphase. Common stochastic gradient-descent (SGD) and a set of PEA approaches, such as follow-the-\nregularized-leader (FTRL), were shown by [36, 14, 43] to be equivalent. Current NAS methods [56,\n55, 30, 25, 2, 46, 23, 17, 29] use Adam, SGD with Momentum or other common optimizers. One\nnotion that is common in PEA principled methods is the regret [4]. PEA strategies aim to guarantee a\nsmall regret under various conditions. We use the regret as a NAS objective, in order to establish a\nbetter principled optimizer than existing methods [23, 35]. Several gradient-descent based optimizers,\nsuch as Adam, present a regret bound analysis, however, the worst-case scenario for Adam has non-\nzero average regret [34], i.e., it is not robust. Our optimizer is designated for selecting architecture\nweights while achieving an optimal regret bound.\n\n6 Conclusion\n\nIn this paper we presented XNAS, a PEA principled optimization method for differential neural\narchitecture search. Inner network architecture weights that govern operations and connections, i.e.\nexperts, are learned via exponentiated-gradient back-propagation update rule. XNAS optimization\ncriterion is well suited for architecture-selection, since it minimizes the regret implied by sub-optimal\nselection of operations with tendency for sparsity, while enabling late bloomers experts to warm-up\nand take over during the search phase. Regret analysis suggests the use of multiple learning rates\nbased on the amount of information carried by the backward gradient. A dynamic mechanism for\nwiping out weak experts is used, reducing the size of computational graph along the search phase,\nhence reducing the search time and increasing the \ufb01nal accuracy. XNAS shows strong performance\non several image classi\ufb01cation datasets, while being among the fastest existing NAS methods.\n\nAcknowledgements\n\nWe would like to thank the members of the Alibaba Israel Machine Vision Lab (AIMVL), in particular\nto Avi Mitrani, Avi Ben-Cohen, Yonathan A\ufb02alo and Matan Protter for their feedbacks and productive\ndiscussions.\n\nReferences\n\n[1] R. Aljundi, P. Chakravarty, and T. Tuytelaars. Expert gate: Lifelong learning with a network of\nexperts. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,\npages 3366\u20133375, 2017.\n\n[2] H. Cai, L. Zhu, and S. Han. Proxylessnas: Direct neural architecture search on target task and\n\nhardware. arXiv preprint arXiv:1812.00332, 2018.\n\n[3] F. P. Casale, J. Gordon, and N. Fusi. Probabilistic neural architecture search. arXiv preprint\n\n[4] N. Cesa-Bianchi and G. Lugosi. Prediction, learning, and games. Cambridge university press,\n\narXiv:1902.05116, 2019.\n\n2006.\n\n[5] K. Chen, L. Xu, and H. Chi. Improved learning algorithms for mixture of experts in multiclass\n\nclassi\ufb01cation. Neural networks, 12(9):1229\u20131252, 1999.\n\n[6] X. Chen, L. Xie, J. Wu, and Q. Tian. Progressive differentiable architecture search: Bridging\n\nthe depth gap between search and evaluation. arXiv preprint arXiv:1904.12760, 2019.\n\n[7] E. D. Cubuk, B. Zoph, D. Mane, V. Vasudevan, and Q. V. Le. Autoaugment: Learning\n\naugmentation policies from data. arXiv preprint arXiv:1805.09501, 2018.\n\n[8] L. N. Darlow, E. J. Crowley, A. Antoniou, and A. J. Storkey. Cinic-10 is not imagenet or\n\ncifar-10. arXiv preprint arXiv:1810.03505, 2018.\n\n[9] T. DeVries and G. W. Taylor. Improved regularization of convolutional neural networks with\n\ncutout. arXiv preprint arXiv:1708.04552, 2017.\n\n[10] D. Eigen, M. Ranzato, and I. Sutskever. Learning factored representations in a deep mixture of\n\nexperts. arXiv preprint arXiv:1312.4314, 2013.\n\n9\n\n\f[11] E. Garmash and C. Monz. Ensemble learning for multi-source neural machine translation. In\nProceedings of COLING 2016, the 26th International Conference on Computational Linguistics:\nTechnical Papers, pages 1409\u20131418, 2016.\n\n[12] I. Goodfellow, Y. Bengio, and A. Courville. Deep learning. MIT press, 2016.\n[13] D. Haussler, J. Kivinen, and M. K. Warmuth. Tight worst-case loss bounds for predicting\nwith expert advice. In European Conference on Computational Learning Theory, pages 69\u201383.\nSpringer, 1995.\n[14] E. Hazan et al. Introduction to online convex optimization. Foundations and Trends R(cid:13) in\n[15] W. Hoeffding. A lower bound for the average sample number of a sequential test. The Annals\n\nOptimization, 2(3-4):157\u2013325, 2016.\n\nof Mathematical Statistics, pages 127\u2013130, 1953.\n\n[16] J. Hu, L. Shen, and G. Sun. Squeeze-and-excitation networks. In Proceedings of the IEEE\n\nconference on computer vision and pattern recognition, pages 7132\u20137141, 2018.\n\n[17] A. Hundt, V. Jain, and G. D. Hager. sharpdarts: Faster and more accurate differentiable\n\narchitecture search. arXiv preprint arXiv:1903.09900, 2019.\n\n[18] R. A. Jacobs, M. I. Jordan, S. J. Nowlan, and G. E. Hinton. Adaptive mixtures of local experts.\n\nNeural Comput., 3(1):79\u201387, Mar. 1991.\n\n[19] P. Jund, N. Abdo, A. Eitel, and W. Burgard. The freiburg groceries dataset. arXiv preprint\n\n[20] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint\n\narXiv:1611.05799, 2016.\n\narXiv:1412.6980, 2014.\n\n[21] J. Kivinen and M. K. Warmuth. Exponentiated gradient versus gradient descent for linear\n\npredictors. information and computation, 132(1):1\u201363, 1997.\n\n[22] G. Larsson, M. Maire, and G. Shakhnarovich. Fractalnet: Ultra-deep neural networks without\n\nresiduals. arXiv preprint arXiv:1605.07648, 2016.\n\n[23] L. Li and A. Talwalkar. Random search and reproducibility for neural architecture search. arXiv\n\npreprint arXiv:1902.07638, 2019.\n\n[24] C. Liu, B. Zoph, M. Neumann, J. Shlens, W. Hua, L.-J. Li, L. Fei-Fei, A. Yuille, J. Huang, and\nK. Murphy. Progressive neural architecture search. In Proceedings of the European Conference\non Computer Vision (ECCV), pages 19\u201334, 2018.\n\n[25] H. Liu, K. Simonyan, and Y. Yang. Darts: Differentiable architecture search. arXiv preprint\n\n[26] I. Loshchilov and F. Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint\n\narXiv:1806.09055, 2018.\n\narXiv:1608.03983, 2016.\n\n[27] R. Luo, F. Tian, T. Qin, E. Chen, and T.-Y. Liu. Neural architecture optimization. In Advances\n\nin Neural Information Processing Systems, pages 7827\u20137838, 2018.\n\n[28] Y. Netzer, T. Wang, A. Coates, A. Bissacco, B. Wu, and A. Y. Ng. Reading digits in nat-\nural images with unsupervised feature learning. In NIPS Workshop on Deep Learning and\nUnsupervised Feature Learning 2011, 2011.\n\n[29] A. Noy, N. Nayman, T. Ridnik, N. Zamir, S. Doveh, I. Friedman, R. Giryes, and L. Zelnik-Manor.\n\nAsap: Architecture search, anneal and prune. arXiv preprint arXiv:1904.04123, 2019.\n\n[30] H. Pham, M. Y. Guan, B. Zoph, Q. V. Le, , and J. Dean. Ef\ufb01cient neural architecture search via\n\nparameter sharing. In International Conference on Machine Learning (ICML), 2018.\n\n[31] N. Qian. On the momentum term in gradient descent learning algorithms. Neural networks,\n\n12(1):145\u2013151, 1999.\n\n[32] C. E. Rasmussen and Z. Ghahramani. In\ufb01nite mixtures of gaussian process experts. In Advances\n\nin neural information processing systems, pages 881\u2013888, 2002.\n\n[33] E. Real, A. Aggarwal, Y. Huang, and Q. V. Le. Regularized evolution for image classi\ufb01er archi-\ntecture search. In International Conference on Machine Learning - ICML AutoML Workshop,\n2018.\n\n[34] S. J. Reddi, S. Kale, and S. Kumar. On the convergence of adam and beyond. arXiv preprint\n\narXiv:1904.09237, 2019.\n\n[35] C. Sciuto, K. Yu, M. Jaggi, C. Musat, and M. Salzmann. Evaluating the search phase of neural\n\n[36] S. Shalev-Shwartz et al. Online learning and online convex optimization. Foundations and\n\narchitecture search. arXiv preprint arXiv:1902.08142, 2019.\nTrends R(cid:13) in Machine Learning, 4(2):107\u2013194, 2012.\n[37] N. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. Le, G. Hinton, and J. Dean. Outra-\ngeously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint\narXiv:1701.06538, 2017.\n\n[38] L. N. Smith. Cyclical learning rates for training neural networks.\n\nIn 2017 IEEE Winter\n\nConference on Applications of Computer Vision (WACV), pages 464\u2013472. IEEE, 2017.\n\n10\n\n\f[39] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and\nA. Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference on\ncomputer vision and pattern recognition, pages 1\u20139, 2015.\n\n[40] C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna. Rethinking the inception archi-\ntecture for computer vision. In Proceedings of the IEEE conference on computer vision and\npattern recognition, pages 2818\u20132826, 2016.\n\n[41] R. Teja Mullapudi, W. R. Mark, N. Shazeer, and K. Fatahalian. Hydranets: Specialized dynamic\narchitectures for ef\ufb01cient inference. In Proceedings of the IEEE Conference on Computer Vision\nand Pattern Recognition, pages 8080\u20138089, 2018.\n\n[42] A. Torralba, R. Fergus, and W. T. Freeman. 80 million tiny images: A large data set for\nnonparametric object and scene recognition. IEEE transactions on pattern analysis and machine\nintelligence, 30(11):1958\u20131970, 2008.\n\n[43] T. Van Erven, W. Kot\u0142owski, and M. K. Warmuth. Follow the leader with dropout perturbations.\n\nIn Conference on Learning Theory, pages 949\u2013974, 2014.\n\n[44] X. Wang, F. Yu, R. Wang, Y.-A. Ma, A. Mirhoseini, T. Darrell, and J. E. Gonzalez. Deep\n\nmixture of experts via shallow embedding. arXiv preprint arXiv:1806.01531, 2018.\n\n[45] R. J. Williams. Simple statistical gradient-following algorithms for connectionist reinforcement\n\nlearning. Machine learning, 8(3-4):229\u2013256, 1992.\n\n[46] B. Wu, X. Dai, P. Zhang, Y. Wang, F. Sun, Y. Wu, Y. Tian, P. Vajda, Y. Jia, and K. Keutzer.\nFbnet: Hardware-aware ef\ufb01cient convnet design via differentiable neural architecture search,\n2018.\n\n[47] H. Xiao, K. Rasul, and R. Vollgraf. Fashion-mnist: a novel image dataset for benchmarking\n\nmachine learning algorithms. arXiv preprint arXiv:1708.07747, 2017.\n\n[48] S. Xie, A. Kirillov, R. Girshick, and K. He. Exploring randomly wired neural networks for\n\nimage recognition. arXiv preprint arXiv:1904.01569, 2019.\n\n[49] S. Xie, H. Zheng, C. Liu, and L. Lin. Snas: stochastic neural architecture search. arXiv preprint\n\narXiv:1812.09926, 2018.\n\n[50] S. Xie, H. Zheng, C. Liu, and L. Lin. Snas: Stochastic neural architecture search. In International\n\nConference on Learning Representations (ICLR), 2019.\n\n[51] B. Yang, G. Bender, Q. V. Le, and J. Ngiam. Soft conditional computation. CoRR,\n\nabs/1904.04971, 2019.\n\n[52] B. Yao, D. Walther, D. Beck, and L. Fei-Fei. Hierarchical mixture of classi\ufb01cation experts\nuncovers interactions between brain regions. In Advances in Neural Information Processing\nSystems, pages 2178\u20132186, 2009.\n\n[53] X. Zhang, Z. Huang, and N. Wang. You only search once: Single shot neural architecture search\n\nvia direct sparse optimization. arxiv 1811.01567, 2018.\n\n[54] Z. Zhong, L. Zheng, G. Kang, S. Li, and Y. Yang. Random erasing data augmentation. arXiv\n\n[55] B. Zoph and Q. V. Le. Neural architecture search with reinforcement learning. arXiv preprint\n\npreprint arXiv:1708.04896, 2017.\n\narXiv:1611.01578, 2016.\n\n[56] B. Zoph, V. Vasudevan, J. Shlens, and Q. V. Le. Learning transferable architectures for scalable\nimage recognition. In Proceedings of the IEEE conference on computer vision and pattern\nrecognition, pages 8697\u20138710, 2018.\n\n11\n\n\f", "award": [], "sourceid": 1164, "authors": [{"given_name": "Niv", "family_name": "Nayman", "institution": "Alibaba Group"}, {"given_name": "Asaf", "family_name": "Noy", "institution": "Alibaba"}, {"given_name": "Tal", "family_name": "Ridnik", "institution": "MIIL Alibaba"}, {"given_name": "Itamar", "family_name": "Friedman", "institution": "Alibaba"}, {"given_name": "Rong", "family_name": "Jin", "institution": "Alibaba"}, {"given_name": "Lihi", "family_name": "Zelnik", "institution": "Alibaba"}]}