{"title": "Efficient Neural Architecture Transformation Search in Channel-Level for Object Detection", "book": "Advances in Neural Information Processing Systems", "page_first": 14313, "page_last": 14322, "abstract": "Recently, Neural Architecture Search has achieved great success in large-scale image classification. In contrast, there have been limited works focusing on architecture search for object detection, mainly because the costly ImageNet pretraining is always required for detectors. Training from scratch, as a substitute, demands more epochs to converge and brings no computation saving.  \n\nTo overcome this obstacle, we introduce a practical neural architecture transformation search(NATS) algorithm for object detection in this paper. Instead of searching and constructing an entire network, NATS explores the architecture space on the base of existing network and reusing its weights. \n\nWe propose a novel neural architecture search strategy in channel-level instead of path-level and devise a search space specially targeting at object detection. With the combination of these two designs, an architecture transformation scheme could be discovered to adapt a network designed for image classification to task of object detection.\n\nSince our method is gradient-based and only searches for a transformation scheme, the weights of models pretrained in ImageNet could be utilized in both searching and retraining stage, which makes the whole process very efficient.\n\nThe transformed network requires no extra parameters and FLOPs, and is friendly to hardware optimization, which is practical to use in real-time application.  \n\nIn experiments, we demonstrate the effectiveness of NATS on networks like {\\em ResNet} and {\\em ResNeXt}. Our transformed networks, combined with various detection frameworks, achieve significant improvements on the COCO dataset while keeping fast.", "full_text": "Ef\ufb01cient Neural Architecture Transformation Search\n\nin Channel-Level for Object Detection\n\nJunran Peng1,2,3 Ming Sun2 Zhaoxiang Zhang1,3\u2217 Tieniu Tan1,3\n\nJunjie Yan2\n\n1University of Chinese Academy of Sciences\n\n2SenseTime Group Limited\n\n3Center for Research on Intelligent Perception and Computing, CASIA\n\nAbstract\n\nRecently, Neural Architecture Search has achieved great success in large-scale\nimage classi\ufb01cation. In contrast, there have been limited works focusing on archi-\ntecture search for object detection, mainly because the costly ImageNet pretraining\nis always required for detectors. Training from scratch, as a substitute, demands\nmore epochs to converge and brings no computation saving. To overcome this\nobstacle, we introduce a practical neural architecture transformation search(NATS)\nalgorithm for object detection in this paper. Instead of searching and constructing\nan entire network, NATS explores the architecture space on the base of existing\nnetwork and reusing its weights. We propose a novel neural architecture search\nstrategy in channel-level instead of path-level and devise a search space specially\ntargeting at object detection. With the combination of these two designs, an archi-\ntecture transformation scheme could be discovered to adapt a network designed for\nimage classi\ufb01cation to task of object detection. Since our method is gradient-based\nand only searches for a transformation scheme, the weights of models pretrained in\nImageNet could be utilized in both searching and retraining stage, which makes the\nwhole process very ef\ufb01cient. The transformed network requires no extra parameters\nand FLOPs, and is friendly to hardware optimization, which is practical to use in\nreal-time application. In experiments, we demonstrate the effectiveness of NATS\non networks like ResNet and ResNeXt. Our transformed networks, combined with\nvarious detection frameworks, achieve signi\ufb01cant improvements on the COCO\ndataset while keeping fast.\n\n1\n\nIntroduction\n\nConvolutional neural networks have achieved signi\ufb01cant success in recent years. With the de-\nvelopment of better optimization and normalization methods [29, 13], many remarkable network\narchitectures [16, 36, 38, 8, 12, 11, 35, 40, 42] have been designed for image classi\ufb01cation based\non hand-crafted heuristics. More recently, great efforts have been taken in neural architecture\nsearch(NAS) that automates the architecture design process, and noticeable results that surpass\nhuman-designed architectures have been reported in image classi\ufb01cation [45, 46, 21, 32, 30, 24, 3].\nHowever, there has been little works that studies NAS on backbone for object detection, mainly for\ntwo reasons: The \ufb01netuning of backbone is always necessary for detectors to converge or achieve a\nhigh performance in short time, otherwise detectors are required to be trained for much more epochs\nwith GN [39] to reach a comparative performance according to [10]. Thus it is inef\ufb01cient to directly\nconduct neural architecture search on object detection. Besides, the essential gap between image\nclassi\ufb01cation and object detection is non-negligible. The experience of NAS in image classi\ufb01cation\ndoes not suf\ufb01ce for NAS in object detection, that the searching space may need to be re-de\ufb01ned.\n\n\u2217Corresponding author.\n\n33rd Conference on Neural Information Processing Systems (NeurIPS 2019), Vancouver, Canada.\n\n\fTable 1: Comparing our method against other NAS methods. The size of training set and input size\nduring search are given to clearly reveal the hardness of searching in different cases. Our ef\ufb01cient\nsearch takes only 20 1080TI GPU days on object detection even though the dataset is of large scale\nand input size is huge.\n\nSize of Train Set\n\nInput Size During Search\n\nGPU-Days\n\nMethods\n\nNASNet [46]\n\nAmoebaNet [32]\nPNASNet [21]\n\nEAS [2]\nDPC [5]\n\nDARTS [24]\n\nProxylessNAS [3]\nAuto-Deeplab [22]\n\nDataset\nCIFAR-10\nCIFAR-10\nCIFAR-10\nCIFAR-10\nCityscapes\nCIFAR-10\nImageNet\nCityscapes\n\nNATS-det\n\nCOCO\n\n50k\n50k\n50k\n50k\n5k\n50k\n1.3M\n5k\n118k\n\n32 \u00d7 32\n32 \u00d7 32\n32 \u00d7 32\n32 \u00d7 32\n769 \u00d7 769\n32 \u00d7 32\n224 \u00d7 224\n321 \u00d7 321\n800 \u00d7 1200\n\n2000\n3000\n150\n10\n2600\n\n4\n10\n3\n20\n\nTask\nCls\nCls\nCls\nCls\nSeg\nCls\nCls\nSeg\nDet\n\nIn this paper, we present effort towards practical meta-learning for object detection task to tackle\nthese two obstacles. Instead of searching an entire network architecture [45, 46, 21, 32, 22, 5], we\nsearch for an architecture transformation strategy that adjusts the structure of existing network to \ufb01t\nthe need of detection, and weights of pretrained model could be fully used in both searching stage\nand re-training stage. As demonstrated in [27], dilation of convolution layers is closely relevant\nto the distribution of ERFs and changing dilation does not in\ufb02uence the kernel size in convolution\nlayer. Therefore a convolution layer with different dilations could reuse the pretrained weights, which\nmakes architecture transformation on dilation-domain possible.\nAdditionally, unlike previous works that search for optimal paths in cell level [46, 21, 32, 30, 24, 3] or\nin network level [45, 31], our transformation search is conducted in channel level. To be speci\ufb01c, we\nsplit the forward signal generated by each path into pieces in channel domain, and treat the sub-paths\nas the minimum searchable units. As shown in Fig. 1, the searched path becomes a fusion of various\noperations with respective channels. With the combination of dilation search space and channel-\nlevel search strategy, our method, named NATS, is able to ef\ufb01ciently discover high-performance\narchitecture transformation scheme for object detection.\nIn our experiments, NATS for detection could improve the AP of Faster-RCNN based on ResNet-50\nand ResNet-101 by 2.0% and 1.8% without any extra parameters or FLOPs, and keep the inference\ntimes almost the same. The transformation is also proved to be valid for various type of detectors.\nOn Mask-RCNN [9], Cascade-RCNN [4] and RetinaNet [20], the AP have been improved by 1.9%,\n1.3% and 1.3% respectively. As shown in Table 1, the searching stage of NATS takes only 2.5 days\non 8 1080TI GPUs, and retraining of searched network takes about 1 day(same as training a baseline\nmodel) with no need of extra pertraining in ImageNet [34], making the whole process ef\ufb01cient and\npractical.\n\n(a)\n\n(b)\n\n(c)\n\nFigure 1: Different forms of operations on edge. In (a), an operation within a connection is chosen on\nthe basis of human heuristics. For path-level search shown in (b), an operation superior to others is\nselected out of all operation candidates. As in (c), the path connecting input and output is decomposed\ninto sub-paths with respective channels.\n\n2\n\n!\u00d7#\u00d7$%&InputOutput'()!\u00d7#\u00d7$*+,Human-Designed!\u00d7#\u00d7$%&InputOutput'()'(*'(+\u2026!\u00d7#\u00d7$,-.Path-wise Selection !\u00d7#\u00d7$%&InputOutput!\u00d7#\u00d7$'()*+,*+-\u2026Channel-wise Selection & Concat *+.*+/\f2 Related works\n\n2.1 Object detection\n\nObject detection is one of the most fundamental \ufb01elds in computer vision for both academic research\nand industrial application. It aims at \ufb01nding the location of each object instances and determining\nthe categories given an image. Some fundamental works like R-CNN [7], Fast-RCNN [6], Faster\nR-CNN [33] and SSD [26] greatly push forward the development of this area. In general, object\ndetectors usually consist of three parts: a backbone that takes in image as input and extract features,\na neck attached to backbone that fuses or further encodes the extracted features and a head for\nclassi\ufb01cation and localization2. In the past years, great progresses have been achieved in designing\neach of these modules.\nFor backbones, there are [17, 37] designed specially for object detection manually. The deformable\nconvolution is also proposed to enable backbone to adaptively sample input features, which is proved\nhelpful in performance but hostile to hardware acceleration. FPN [19] is one of the representative\nwork exploring the architecture of neck. It builds a top-down structure with lateral connections to\ndifferent stages of backbone to integrate features at all scales. Many recent works [15, 14, 41] propose\nvarious multi-scale integration strategies to generate pyramidal feature representations.\nIn [25, 27], it is proposed that the effective receptive \ufb01elds(ERFs) of backbones is essential for object\ndetection and dilation of convolutions could effectively change the distribution of ERFs. Based on\nthese \ufb01ndings, we aim to design a network architecture that holds better ERFs to handle the huge\nvariation of object scales in detection.\n\n2.2 Neural architecture search\n\nDesigning network automatically has drawn great attention recently. Several works [46, 45, 2, 1, 43]\nintroduce reinforcement learning with RNN controller to design cell structure to form a network.\nIn [32, 23, 28], evolution method has been used to update network structure instead of RL-based\ncontroller. These methods are sample-based that often take great amount of computational resources.\nIn [30, 2], weights of sampled models could be reused to reduce the search cost.\nSome other works tend to used gradient-based methods that search for relatively optimal child net-\nworks from prede\ufb01ned super-nets, which make NAS with limited computational resources possible.\nDARTS [24] formulates a super-network based on the continuous relaxation of the architecture repre-\nsentation, which allows ef\ufb01cient search of the architecture using gradient descent. ProxylessNAS [3]\nfurther improves the optimization strategy and imports latency loss to \ufb01nd more ef\ufb01cient architectures.\nIn Auto-DeepLab [22], gradient-based method is also applied to search for backbone of segmentation\nmodel.\nAs for search space, most methods tend to search for optimal paths in cell level [46, 21, 32, 30, 24, 3]\nor network level [45, 31], while in this paper, we propose a novel search space in channel level.\nInspired by the idea of function-preserving transformation in [2], we propose a neural architecture\ntransformation search(NATS) algorithm to automatically \ufb01nd an optimal strategy to transform the\nstructure of existing networks designed for image classi\ufb01cation to \ufb01t task of object detection.\n\n3 Methods\n\nIn this section, we \ufb01rst analyze a crucial factor of backbone for object detection. Then we describe\nour general strategy of neural architecture search in channel-domain and design a search space that\nenables effective architecture transformation search specialized in object detection.\n\n3.1 Revisit effective receptive \ufb01elds\n\nReceptive \ufb01eld is one of the most basic concepts in deep CNNs. Unlike in fully connected networks\nthat value of each neuron is associated with entire input to network, a neuron in convolutional\nnetworks depends on a certain region of the input. This property enables neurons in convolutional\nnetworks to be position-sensitive, and makes dense prediction tasks like object detection and semantic\n\n2For one-stage detectors, the form of head is fully convolution.\n\n3\n\n\fFigure 2: The structure of block during search. The output of each operation is equally divided into\nsub-groups in channel domain. Each sub-group of each candidate is assigned an architecture parameter\nto \ufb01t together as output, which makes the search space within each channel group continuous. The\nsearch between channel groups is independent.\n\nsegmentation possible. As carefully studied in [27], the distribution of impact in a receptive \ufb01eld is\nproved to be like a Gaussian and only a small central region of pixels in receptive \ufb01eld effectively\ncontributes to response of neuron in output map. The region is called effective receptive \ufb01eld(ERF).\nIn tasks of image classi\ufb01cation, the input sizes are always kept small. As in object detection, the input\nsizes are often much bigger and detectors are required to handle objects over a large range of scales,\nthus the ERFs of network designed for image classi\ufb01cation could not suf\ufb01ce for this demand3. As\nmentioned in [27], changing dilations could effectively modify the ERFs distribution of convolution\nlayers. Moreover, changing dilation does not in\ufb02uence the kernel size of convolution layer, which\nenables pretrained weights to be directly reused. Therefore in this work, we constrain our search\nspace to dilations of convolution layers in order to grant network better ERFs for handling the huge\nvariation of object scales.\n\n3.2 Channel-level neural architecture search\n\nA neural network is a directed acyclic graph consisting of a set of nodes connected in order. The\ndirected edges connecting nodes are always associated with some operations that process the input\nsignals, such as convolutional layer, max-pooling and etc. For most gradient-based NAS methods,\nan over-parameterized super-network is constructed \ufb01rstly with all candidates paths included and\none superior path is selected on each edge with the other candidates removed. However, signals in\nnetwork often contain numerous channels during forward propagation, which means that a path is not\nthe minimum separable structure unit in network and path-level search methods [3, 24, 22] limit the\ngranularity of architecture search. Thus in our work, we treat a channel of signal generated by an\noperation of certain genotype as the minimum separable structure unit, and transform path-level NAS\ninto channel-level NAS.\nGiven an input signal x, the output signal y\u2217 is generated based on the outputs of all G candidate\npaths during search. Each path is associated with a certain type of operation Og, and we call the\ncategories of operation as genotypes G with g \u2208 G. While in DARTS and Auto-Deeplab, each entire\npath is assigned an architecture parameter \u03b1g and y\u2217 is weighted sum of input signals where the\nweights are calculated by applying softmax to \u03b1g:\ny\u2217 =\n\nyg = Og(x),\n\n(cid:88)\n\nyg\n\n(1)\n\n(cid:80)\n\ng\u2208G\n\nexp(\u03b1g)\ng(cid:48)\u2208G exp(\u03b1g(cid:48)\n\n)\n\nAfter obtaining the continuous super-architecture with \u03b1, every edge with mixed operation of all\ngenotypes is replaced with the most likely operation by taking the argmax of \u03b1g. Thus only one\ngenotype is selected to handle input signals on each edge in the outcome architecture.\nTo apply a more \ufb01ne-grained architecture search, we equally divide yg into N groups in channel\ndomain for each genotype as follows:\n\nN(cid:88)\n\nyg \u21d2 {yg\n\n1 , yg\n\n2 , ..., yg\n\ni , ..., yg\n\nN}, with Cout =\n\nC g\ni ,\n\n(2)\n\n3Given a conventional input size of 800 \u00d7 1200, size of objects varies from 32 to 800 pixels in COCO, while\n\nthe size of ERFs in ResNet50 is approximately 100 pixels as shown in 4(a).\n\ni\n\n4\n\n!\"#,$%&'=$!\"),$%&'=$!\"*,$%&'=$!\"+,$%&'=$\u2026Weighted SumBN, ReLUInputOutput\u2026Channel Groups,##,)#,*#,+#,#),)),*),+)\fFigure 3: Decoding discrete architecture based on intensity of genotypes.\n\nwhere i denotes the index of channel group and Cout denotes the total output channels. As illustrated\nin Fig. 2, instead of assigning path-wise architecture parameters we assign each channel group an\ni where 1 \u2264 i \u2264 N. We use the continuous relaxation among genotypes in\narchitecture parameter \u03b1g\neach channel group and the output of group i is obtained as:\n\n(cid:88)\n\ng\u2208G\n\ny\u2217\ni =\n\n(cid:80)\nexp(ag\ni )\ng(cid:48)\u2208G exp(ag(cid:48)\ni )\n\nyg\ni\n\n(3)\n\nIn this way, the super-net is constructed in which nodes are connected with sub-paths in channel\ndomain and architecture parameters \u03b1g\ni are learnt for each genotype in each channel group. The\ntraining set is divided into two splits, and the optimization alternates between updating network\nparameters in the \ufb01rst split and updating architecture parameters \u03b1g\n\ni in the other split.\n\n3.3 Decoding discrete architectures with channel decomposition\n\nUnlike [24], [22] and [3]that select path with the maximum probability and prune redundant paths,\nthe discrete architecture decoding in our method is conducted based on the distribution of \u03b1g\ni . We\n\ufb01rst keep the index of genotype with the maximum probability in each channel group as\n\nand calculate the intensity of each genotype throughout all channel groups as:\n\nindi = arg max\n\n\u03b1g\ni ,\n\n(cid:80)N\n\ng\n\nN\n\nI g =\n\ni 1(indi = g)\n\n(4)\n\n(5)\n\nAs illustrated in Fig. 3, we retain all the paths that have a positive I g but reset output channels\naccording to I g as C g\nout = CoutI g. The output feature maps of different genotypes are concatenated\ntogether to form a \ufb01nal output y as follows:\n\n{y1, y2, ..., yg, ..., yG} \u21d2 y\n\n(6)\n\n3.4 Architecture transformation search for object detection\n\nTaking bottleneck structure in ResNet as example in our paper, the transformation search is applied\non the 3 \u00d7 3 convolution layer in the middle. Dilations in both orientation of the convolution\n{dh, dw} is set as our search space. Since changing dilations does not modify the kernel size or\nthe shape of weights, we could directly transfer weights of pretrained model to our networks in\nboth searching stage and retraining stage Combining the channel-domain searching strategy with\nthe dilation search space makes our neural architecture transformation search possible. The whole\nprocess is gradient-based and extra pretraining is of no need, which makes our method very ef\ufb01cient.\nDuring the training of super-network, backbone is initialized with the weights pretrained on ImageNet.\nFor each 3 \u00d7 3 convolution layer in stage-3,4,5, weights are copied to all of its dilated replicas.\nWeight initialization for searched model is different in re-training stage. Since the original 3 \u00d7 3\nconvolution layer has been decomposed into sub-convs with various dilations and output channels,\nthe pretrained weight W with shape Cout \u00d7 Cin \u00d7 K \u00d7 K is also decomposed into G groups with\nshape {C g\n\ng=1 in order to \ufb01t the weights shape of sub-convs.\n\nout \u00d7 Cin \u00d7 K \u00d7 K}G\n\n5\n\n!\"#,$%&'=)#$!\"*,$%&'=)*$!\"+,$%&'=)+$!\",,$%&'=),$\u2026ConcatBN, ReLUInputOutputOp1Op2Op3...OpG00.20.40.60.81IgIntensity of GenotypesI1I2I3...IG\fTable 2: Performance on minival with \ufb01xed number of channel groups for NATS. When number of\ngroups is 1, the architecture transformation search is on path-level.\n\nNum of Groups\n\nbaseline\n\n1\n2\n4\n8\n16\n32\n\nAP\n36.4\n36.9\n37.2\n37.9\n37.8\n38.4\n38.2\n\nAP50\n58.9\n58.9\n59.6\n60.2\n60.4\n61.0\n60.6\n\nAP75\n38.9\n39.1\n39.8\n40.9\n40.4\n41.2\n41.0\n\nAPS\n21.4\n21.3\n21.6\n22.2\n21.4\n22.5\n22.3\n\nAPM\n39.8\n40.1\n40.8\n40.9\n41.3\n41.8\n41.7\n\nAPM\n39.8\n41.4\n41.6\n41.6\n41.9\n40.9\n\nAPL\n47.2\n47.5\n48.9\n49.9\n50.0\n50.4\n50.1\n\nAPL\n47.2\n50.3\n50.2\n50.5\n50.4\n50.3\n\nTable 3: Performance on minival with \ufb01xed number of channels per group for NATS.\n\nAP\n36.4\n38.0\n38.1\n38.2\n38.3\n37.8\n\nAP50\n58.6\n60.5\n60.7\n60.7\n60.9\n60.5\n\nAP75\n38.6\n40.5\n40.7\n40.9\n41.3\n40.4\n\nAPS\n21.0\n22.5\n22.3\n22.4\n22.3\n21.7\n\nChannels Per Group\n\nbaseline\n\n1\n8\n16\n32\n64\n\n4 Experiments and results\n\n4.1 COCO dataset\n\nWe use the MS-COCO [18] for experiment in this paper. It contains 83K training images in train2014\nand 40K validation images in val2014. In its 2017 version, it has 118K images in train2017 set and\n5K images in val2017(a.k.a minival). The dataset is widely believed challenging in particular due\nto huge variation of object scales and large number of objects per image. We consider AP@IoU\nas evaluation metric which averages mAP across IoU threshold ranging from 0.50 to 0.95 with an\ninterval of 0.05. During searching stage, we use train2014 for training model parameters and use 35K\nimages from val2014 that are not in minival for calibrating architecture parameters. During retraining\nstage, our searched model is trained with train2017 and evaluated with minival as convention.\n\n4.2\n\nImplementation details\n\nIn our method we \ufb01rstly search for an appropriate structure transformation scheme on COCO2014\ndataset, then we train our searched model on COCO2017 dataset as mentioned above. We experiment\non the Faster-RCNN baselines with FPN [19], and adopt models pretrained in ImageNet [34] for\nweight initialization in both searching and training stages.\n\nSearching details. We conduct architecture transformation search for 25 epochs in total. To make\nthe super-network converge better, architecture parameters are designed not to be updated in the \ufb01rst\n10 epochs. The batch size is 1 image per GPU due to GPU memory constraint. We use SGD optimizer\nwith momentum 0.9 and weight decay 0.0001 for training model weights. Cosine annealing learning\nrate that decays from 0.00125 to 0.00005 is applied as lr-scheduler. When training architecture\nparameters \u03b1, we use Adam optimizer with learning rate 0.01 and weight decay 0.00001.\n\nTraining details. After the architecture searching is \ufb01nished, we decode discrete architecture as\nmentioned in 3.3. We use SGD optimizer with 0.9 momentum and 0.0001 weight decay. For fair\ncomparison, all our model is trained for 13 epochs, known as 1\u00d7 schedule. The initial learning rate\nis set 0.00125 per image and is divided by 10 at 8 and 11 epochs. Warming up and Synchronized\nBatchNorm mechanism are applied in both baselines and our searched models for multi-GPU training.\nIt takes approximately 2.5 days to \ufb01nish the search for 8 1080TI GPUs.\n\n4.3 Object detection results\n\nIn our paper, ResNet[8] and ResNeXt[40] are selected as backbone in all experiment settings.\nFollowing the regime mentioned in DCNv2 [44], we apply architecture transformation search only\n\n6\n\n\fTable 4: Performance of NATS on ResNet101 and ResNeXt101. NATS is conducted with \ufb01xed\nnumber of channel per group as C = 32 in this ablation study.\n\nAP75\n41.7\n44.0\n44.1\n45.2\n\nAPS\n22.8\n23.2\n24.2\n24.9\n\nAPM\n42.8\n44.1\n45.1\n45.5\n\nBackbone\n\nR101\n\nR101-NATS\nX101-32\u00d74d\n\nX101-32\u00d74d-NATS\n\nAP\n38.6\n40.4\n40.5\n41.6\n\nAP50\n60.7\n62.6\n63.1\n64.3\n\nAPL\n49.6\n53.3\n52.9\n54.8\n\nTable 5: Comparison between different sets of genotypes on COCO minival.\n\nGENOs\nbaseline\nNATS-A\nNATS-B\nNATS-C\n\nAP\n36.4\n37.7\n38.0\n38.4\n\nAP50\n58.9\n59.9\n60.5\n61.0\n\nAP75\n38.9\n40.5\n40.7\n41.2\n\nAPS\n21.4\n22.0\n21.8\n22.5\n\nAPM\n39.8\n40.8\n41.4\n41.8\n\nAPL\n47.2\n49.8\n50.5\n50.4\n\non blocks of stage-3,4,5 in backbone. For stage-3 and stage-4, the dilation candidates are {1, 2, 3,\n(1,3), (3,1)}. The dilation candidates of stage-5 are {1, 2, 3, 4, 5, (1,3), (3,1), (1,5), (5,1)}. No extra\nparameters or FLOPs is imported in our transformed architectures.\n\nGroup division. We evaluate different ways of dividing output channels into groups. With a given\n\ufb01xed group number(G \u2208 {1, 2, 4, 8, 16, 32}, NATS is applied on ResNet-50. In Table 2, we \ufb01nd that\nmore groups could achieve better performance. With a \ufb01xed group number of 16, the transformed\narchitecture achieves an AP of 38.4%(2.0% higher than the baseline). Note that G = 1 is a special\ncase which is the path-level searching strategy similar to DARTS [24] and ProxylessNAS [3], and the\nimprovement is limited(only 0.5% over baseline).\nWe also \ufb01xed the number of channels(C \u2208 {1, 8, 16, 32, 64} per searching group. Since different\nblocks have different channel numbers, the group number can change across layers in this setting.\nThe results are shown in Table 3. With a \ufb01xed channel number per searching group, our transformed\nResNet-50 achieves a minival AP of 38.3% which is 1.9% higher than baselines.\nFrom both setting we \ufb01nd that searching with a more \ufb01ne-grained grouping is better in general. We\ninfer that it enables blocks to have more combinations of operations with different dilations, which\nbrings more \ufb02exible ERFs. We also \ufb01nd that the improvement of AP increases as scale of objects\ngrows. In model searched with G = 16, the improvements of APS, APM and APL are 1.5%, 2.0%\nand 3.2% respectively.\n\nDeeper models.\nIt is known that deeper networks have larger ERFs with stronger intensity and\nmay dilute the effects of many approaches, thus we study the impact of architecture transformation\non deeper networks. We have compared transformed backbones with baselines on ResNet-101 and\nResNeXt-101. As shown in table 4, architecture transformation yields 1.8% AP improvement on\nResNet-101, from 38.6 to 40.4. While in ResNeXt-101, we use the 32 \u00d7 4d con\ufb01guration and the\nchannels per group is set 32 to be consistent with backbone. Architecture transformation yields 1.1\nimprovement from 40.5 to 41.6. Comparing ResNet-101 with shallower network like ResNet-50,\nthe improvement of APS is relatively small(0.4% v.s. 0.7%), but improvement of APL is even\ngreater(3.7% v.s. 3.2%) even through it is deeper. ResNeXt-101 acts also in the similar way.\n\nIn\ufb02uence of genotypes.\nIn this section we explore the in\ufb02uence of genotypes included during\narch-transformation search. We include different set of dilation candidates as genotypes in this\nablation study. We \ufb01rst investigate the necessity of dense dilation candidates. For stage-3,4,5 we\nset the dilation candidates {1, 3}, {1, 3}, {1, 3, 5} respectively as setting A, and set the dilation\ncandidates {1, 2, 3}, {1, 2, 3}, {1, 2, 3, 4, 5} as setting B. To explore the in\ufb02uence of ratio aspects we\nadd (1, 3),(3, 1) for stage-3,4 and (1, 3), (3, 1), (1, 5), (5, 1) for stage-5 as candidates in setting C.\nAs shown in Table 5, NATS-B is higher than PATS-A by 0.3%, which implies that denser dilation\ncandidates is slightly better. NATS-C is 0.4% better than PATS-B, demonstrating that dilations with\naspect ratios are bene\ufb01cial for object detection.\n\n7\n\n\fTable 6: Comparison of performance of NATS on different type of detectors.\n\nMethod\n\nBackbone\n\nFaster-RCNN\nFaster-RCNN\nMask-RCNN\nMask-RCNN\nCascade-RCNN\nCascade-RCNN\n\nRetinaNet\nRetinaNet\n\nR50-NATS\n\nR50\n\nR50\n\nR50-NATS\n\nR50-NATS\n\nR50\n\nR50\n\nR50-NATS\n\nAP\n36.4\n38.4\n37.5\n39.3\n40.7\n42.0\n36.0\n37.3\n\nAP50\n58.9\n61.0\n59.6\n61.3\n59.4\n61.4\n56.1\n57.8\n\nAP75\n38.9\n40.8\n40.5\n42.6\n44.2\n45.5\n38.6\n39.5\n\nAPS\n21.4\n22.1\n22.0\n23.0\n22.9\n24.2\n20.4\n20.7\n\nAPM\n39.8\n41.5\n41.0\n42.5\n43.9\n45.3\n40.0\n40.8\n\nAPL\n47.2\n50.5\n48.4\n51.7\n54.2\n55.9\n48.2\n49.6\n\n(a) R50.\n\n(b) NATS-R50.\n\n(c) R101.\n\n(d) NATS-R101.\n\nFigure 4: Visualization of ERFs in transformed architectures and vanilla architectures.\n\nVarious detectors.\nTo validate the generalization ability of our method, we also combine the\ntransformed networks with different type of detectors. Several well-known and remarkable frame-\nworks like Mask-RCNN [9], Cascade-RCNN [4] and RetinaNet [20] are selected in this ablation\nstudy. ResNet-50 and the transformed ResNet-50(G=16) are selected as backbones and all the models\nare trained with 1\u00d7 lr-schedule. As demonstrated in Table 6, performances of all detectors in chart\nare improved prominently(1.8% in Mask-RCNN, 1.3% in Cascade-RCNN and 1.3% in RetinaNet).\nThis shows the strong generalization capability of networks searched through our transformation\nmethod.\n\n4.4 Visualization of ERFs\n\nFollowing the regime mentioned in [27], we visualize the receptive \ufb01eld of neuron on map of the\nlast convolution layer. The input values are set 1 for whole image and only the neuron in the center\nof output map propagates backward. To focus on only the intensity of connections, ReLUs are\nabandoned during visualization. As shown in the Fig. 4, the size of ERFs in our transformed network\nare larger than ERFs of the vanilla structures. While the intensities in the center region are kept\nstrong, the intensities of outer region becomes weaker as the region becomes bigger. It indicates that\nthis type of ERFs could better \ufb01t the task of object detection.\n\n5 Conclusion\n\nIn this paper, we present NATS that can ef\ufb01ciently learn an neural architecture transformation strategy\nto adapt existing networks to new tasks. We propose a novel architecture search scheme in channel\ndomain and design a search space of dilations targeting at object detection, which makes the neural\narchitecture transformation search possible. Finetuning from pretrained models is feasible in both\nsearching and re-training stages, making the whole process very ef\ufb01cient. Experiments on the COCO\ndataset have demonstrated that NATS could effectively improve the capability of networks to handle\nhuge variation of object scales and robustly yield improvements on various type of detectors. In the\nfuture, we would like to investigate architecture transformation search on depth and width of each\nstage for object detection task.\n\n8\n\n050100150200250300350400050100150200250300350400050100150200250300350400050100150200250300350400050100150200250300350400050100150200250300350400050100150200250300350400050100150200250300350400\f6 Acknowledgements\n\nThis work was supported in part by the National Key R&D Program of China(No.2018YFB-1402605),\nthe Beijing Municipal Natural Science Foundation (No.Z181100008918010), the National Natural\nScience Foundation of China(No.61836014, No.61761146004, No.61773375, No.61602481) and\nCAS-AIR.\n\nReferences\n[1] B. Baker, O. Gupta, N. Naik, and R. Raskar. Designing neural network architectures using reinforcement\n\nlearning. In ICLR, 2017.\n\n[2] H. Cai, T. Chen, W. Zhang, Y. Yu, and J. Wang. Ef\ufb01cient architecture search by network transformation.\n\nIn AAAI, 2018.\n\n[3] H. Cai, L. Zhu, and S. Han. Proxylessnas: Direct neural architecture search on target task and hardware.\n\narXiv preprint arXiv:1812.00332, 2018.\n\n[4] Z. Cai and N. Vasconcelos. Cascade r-cnn: Delving into high quality object detection. In Proceedings of\n\nthe IEEE Conference on Computer Vision and Pattern Recognition(CVPR), 2018.\n\n[5] L.-C. Chen, M. Collins, Y. Zhu, G. Papandreou, B. Zoph, F. Schroff, H. Adam, and J. Shlens. Searching\n\nfor ef\ufb01cient multi-scale architectures for dense image prediction. In NIPS, 2018.\n\n[6] R. Girshick. Fast r-cnn. In ICCV, 2015.\n\n[7] R. Girshick, J. Donahue, T. Darrell, and J. Malik. Rich feature hierarchies for accurate object detection\n\nand semantic segmentation. In CVPR, 2014.\n\n[8] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In CVPR, 2016.\n\n[9] K. He, G. Gkioxari, P. Doll\u00e1r, and R. Girshick. Mask r-cnn. In ICCV, 2017.\n\n[10] K. He, R. Girshick, and P. Doll\u00e1r. Rethinking imagenet pre-training. arXiv preprint arXiv:1811.08883,\n\n2018.\n\n[11] J. Hu, L. Shen, and G. Sun. Squeeze-and-excitation networks. In CVPR, 2018.\n\n[12] G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger. Densely connected convolutional networks.\n\nIn CVPR, 2017.\n\n[13] S. Ioffe and C. Szegedy. Batch normalization: Accelerating deep network training by reducing internal\n\ncovariate shift. In ICML, 2015.\n\n[14] T. Kong, F. Sun, A. Yao, H. Liu, M. Lu, and Y. Chen. Ron: Reverse connection with objectness prior\n\nnetworks for object detection. In CVPR, 2017.\n\n[15] T. Kong, F. Sun, C. Tan, H. Liu, and W. Huang. Deep feature pyramid recon\ufb01guration for object detection.\n\nIn ECCV, 2018.\n\n[16] A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classi\ufb01cation with deep convolutional neural\n\nnetworks. In NIPS, 2012.\n\n[17] Z. Li, C. Peng, G. Yu, X. Zhang, Y. Deng, and J. Sun. Detnet: Design backbone for object detection. In\n\nECCV, 2018.\n\n[18] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Doll\u00e1r, and C. L. Zitnick. Microsoft\n\ncoco: Common objects in context. In ECCV, 2014.\n\n[19] T.-Y. Lin, P. Doll\u00e1r, R. Girshick, K. He, B. Hariharan, and S. Belongie. Feature pyramid networks for\n\nobject detection. In CVPR, 2017.\n\n[20] T.-Y. Lin, P. Goyal, R. Girshick, K. He, and P. Doll\u00e1r. Focal loss for dense object detection. In ICCV, 2017.\n\n[21] C. Liu, B. Zoph, M. Neumann, J. Shlens, W. Hua, L.-J. Li, L. Fei-Fei, A. Yuille, J. Huang, and K. Murphy.\n\nProgressive neural architecture search. In ECCV, 2018.\n\n[22] C. Liu, L.-C. Chen, F. Schroff, H. Adam, W. Hua, A. Yuille, and L. Fei-Fei. Auto-deeplab: Hierarchical\n\nneural architecture search for semantic image segmentation. arXiv preprint arXiv:1901.02985, 2019.\n\n9\n\n\f[23] H. Liu, K. Simonyan, O. Vinyals, C. Fernando, and K. Kavukcuoglu. Hierarchical representations for\n\nef\ufb01cient architecture search. In ICLR, 2018.\n\n[24] H. Liu, K. Simonyan, and Y. Yang. Darts: Differentiable architecture search.\n\narXiv:1806.09055, 2018.\n\narXiv preprint\n\n[25] S. Liu, D. Huang, et al. Receptive \ufb01eld block net for accurate and fast object detection. In ECCV, 2018.\n\n[26] W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C.-Y. Fu, and A. C. Berg. Ssd: Single shot multibox\n\ndetector. In ECCV, 2016.\n\n[27] W. Luo, Y. Li, R. Urtasun, and R. Zemel. Understanding the effective receptive \ufb01eld in deep convolutional\n\nneural networks. In NIPS, 2016.\n\n[28] R. Miikkulainen, J. Liang, E. Meyerson, A. Rawal, D. Fink, O. Francon, B. Raju, H. Shahrzad,\nA. Navruzyan, N. Duffy, et al. Evolving deep neural networks. In Arti\ufb01cial Intelligence in the Age\nof Neural Networks and Brain Computing, pages 293\u2013312. Elsevier, 2019.\n\n[29] V. Nair and G. E. Hinton. Recti\ufb01ed linear units improve restricted boltzmann machines. In ICML, 2010.\n\n[30] H. Pham, M. Y. Guan, B. Zoph, Q. V. Le, and J. Dean. Ef\ufb01cient neural architecture search via parameter\n\nsharing. arXiv preprint arXiv:1802.03268, 2018.\n\n[31] E. Real, S. Moore, A. Selle, S. Saxena, Y. L. Suematsu, J. Tan, Q. V. Le, and A. Kurakin. Large-scale\n\nevolution of image classi\ufb01ers. In ICML, 2017.\n\n[32] E. Real, A. Aggarwal, Y. Huang, and Q. V. Le. Regularized evolution for image classi\ufb01er architecture\n\nsearch. arXiv preprint arXiv:1802.01548, 2018.\n\n[33] S. Ren, K. He, R. Girshick, and J. Sun. Faster r-cnn: Towards real-time object detection with region\n\nproposal networks. In NIPS, 2015.\n\n[34] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla,\n\nM. Bernstein, et al. Imagenet large scale visual recognition challenge. IJCV, 2015.\n\n[35] M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L.-C. Chen. Mobilenetv2: Inverted residuals and\n\nlinear bottlenecks. In CVPR, 2018.\n\n[36] K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. In\n\nICLR, 2015.\n\n[37] S. Sun, J. Pang, J. Shi, S. Yi, and W. Ouyang. Fishnet: A versatile backbone for image, region, and pixel\n\nlevel prediction. In NIPS, 2018.\n\n[38] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich.\n\nGoing deeper with convolutions. In CVPR, 2015.\n\n[39] Y. Wu and K. He. Group normalization. In ECCV, 2018.\n\n[40] S. Xie, R. Girshick, P. Doll\u00e1r, Z. Tu, and K. He. Aggregated residual transformations for deep neural\n\nnetworks. In CVPR, 2017.\n\n[41] S. Zhang, L. Wen, X. Bian, Z. Lei, and S. Z. Li. Single-shot re\ufb01nement neural network for object detection.\n\nIn Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition(CVPR), 2018.\n\n[42] X. Zhang, X. Zhou, M. Lin, and J. Sun. Shuf\ufb02enet: An extremely ef\ufb01cient convolutional neural network\n\nfor mobile devices. In CVPR, 2018.\n\n[43] Z. Zhong, J. Yan, W. Wu, J. Shao, and C.-L. Liu. Practical block-wise neural network architecture\n\ngeneration. In CVPR, 2018.\n\n[44] X. Zhu, H. Hu, S. Lin, and J. Dai. Deformable convnets v2: More deformable, better results. arXiv preprint\n\narXiv:1811.11168, 2018.\n\n[45] B. Zoph and Q. V. Le. Neural architecture search with reinforcement learning. In ICLR, 2017.\n\n[46] B. Zoph, V. Vasudevan, J. Shlens, and Q. V. Le. Learning transferable architectures for scalable image\n\nrecognition. In CVPR, 2018.\n\n10\n\n\f", "award": [], "sourceid": 8079, "authors": [{"given_name": "Junran", "family_name": "Peng", "institution": "CASIA"}, {"given_name": "Ming", "family_name": "Sun", "institution": "sensetime.com"}, {"given_name": "ZHAO-XIANG", "family_name": "ZHANG", "institution": "Chinese Academy of Sciences, China"}, {"given_name": "Tieniu", "family_name": "Tan", "institution": "Chinese Academy of Sciences"}, {"given_name": "Junjie", "family_name": "Yan", "institution": "Sensetime Group Limited"}]}