{"title": "SNIPER: Efficient Multi-Scale Training", "book": "Advances in Neural Information Processing Systems", "page_first": 9310, "page_last": 9320, "abstract": "We present SNIPER, an algorithm for performing efficient multi-scale training in instance level visual recognition tasks. Instead of processing every pixel in an image pyramid, SNIPER processes context regions around ground-truth instances (referred to as chips) at the appropriate scale. For background sampling, these context-regions are generated using proposals extracted from a region proposal network trained with a short learning schedule. Hence, the number of chips generated per image during training adaptively changes based on the scene complexity. SNIPER only processes 30% more pixels compared to the commonly used single scale training at 800x1333 pixels on the COCO dataset. But, it also observes samples from extreme resolutions of the image pyramid, like 1400x2000 pixels. As SNIPER operates on resampled low resolution chips (512x512 pixels), it can have a batch size as large as 20 on a single GPU even with a ResNet-101 backbone. Therefore it can benefit from batch-normalization during training without the need for synchronizing batch-normalization statistics across GPUs. SNIPER brings training of instance level recognition tasks like object detection closer to the protocol for image classification and suggests that the commonly accepted guideline that it is important to train on high resolution images for instance level visual recognition tasks might not be correct. Our implementation based on Faster-RCNN with a ResNet-101 backbone obtains an mAP of 47.6% on the COCO dataset for bounding box detection and can process 5 images per second during inference with a single GPU. Code is available at https://github.com/MahyarNajibi/SNIPER/ .", "full_text": "SNIPER: Ef\ufb01cient Multi-Scale Training\n\nBharat Singh \u2217\n\nMahyar Najibi \u2217\n\nUniversity of Maryland, College Park\n{bharat,najibi,lsd}@cs.umd.edu\n\nLarry S. Davis\n\nAbstract\n\nWe present SNIPER, an algorithm for performing ef\ufb01cient multi-scale training in\ninstance level visual recognition tasks. Instead of processing every pixel in an\nimage pyramid, SNIPER processes context regions around ground-truth instances\n(referred to as chips) at the appropriate scale. For background sampling, these\ncontext-regions are generated using proposals extracted from a region proposal\nnetwork trained with a short learning schedule. Hence, the number of chips gener-\nated per image during training adaptively changes based on the scene complexity.\nSNIPER only processes 30% more pixels compared to the commonly used single\nscale training at 800x1333 pixels on the COCO dataset. But, it also observes\nsamples from extreme resolutions of the image pyramid, like 1400x2000 pixels. As\nSNIPER operates on resampled low resolution chips (512x512 pixels), it can have a\nbatch size as large as 20 on a single GPU even with a ResNet-101 backbone. There-\nfore it can bene\ufb01t from batch-normalization during training without the need for\nsynchronizing batch-normalization statistics across GPUs. SNIPER brings training\nof instance level recognition tasks like object detection closer to the protocol for\nimage classi\ufb01cation and suggests that the commonly accepted guideline that it is\nimportant to train on high resolution images for instance level visual recognition\ntasks might not be correct. Our implementation based on Faster-RCNN with a\nResNet-101 backbone obtains an mAP of 47.6% on the COCO dataset for bounding\nbox detection and can process 5 images per second during inference with a single\nGPU. Code is available at https://github.com/mahyarnajibi/SNIPER/.\n\n1\n\nIntroduction\n\nHumans have a foveal visual system which attends to objects at a \ufb01xed distance and size. For\nexample, when we focus on nearby objects, far away objects get blurred [7]. Naturally, it is dif\ufb01cult\nfor us to focus on objects of different scales simultaneously [30]. We only process a small \ufb01eld of\nview at any given point of time and adaptively ignore the remaining visual content in the image.\nHowever, computer algorithms which are designed for instance level visual recognition tasks like\nobject detection depart from this natural way of processing visual information. For obtaining a\nrepresentation robust to scale, popular detection algorithms like Faster-RCNN/Mask-RCNN [29, 12]\nare trained on a multi-scale image pyramid [22, 33]. Since every pixel is processed at each scale,\nthis approach to processing visual information increases the training time signi\ufb01cantly. For example,\nconstructing a 3 scale image pyramid (e.g. scales=1x,2x,3x) requires processing 14 times the number\nof pixels present in the original image. For this reason, it is impractical to use multi-scale training in\nmany scenarios.\nRecently, it is shown that ignoring gradients of objects which are of extreme resolutions is bene\ufb01cial\nwhile using multiple scales during training [33]. For example, when constructing an image pyramid\nof 3 scales, the gradients of large and small objects should be ignored at large and small resolutions\n\n\u2217Equal Contribution\n\n32nd Conference on Neural Information Processing Systems (NeurIPS 2018), Montr\u00e9al, Canada.\n\n\frespectively. If this is the case, an intuitive question which arises is, do we need to process the entire\nimage at a 3x resolution? Wouldn\u2019t it suf\ufb01ce to sample a much smaller RoI (chip) around small\nobjects at this resolution? On the other hand, if the image is already high resolution, and objects in it\nare also large in size, is there any bene\ufb01t in upsampling that image?\nWhile ignoring signi\ufb01cant portions of the image would save computation, a smaller chip would also\nlack context required for recognition. A signi\ufb01cant portion of background would also be ignored\nat a higher resolution. So, there is a trade-off between computation, context and negative mining\nwhile accelerating multi-scale training. To this end, we present a novel training algorithm called\nScale Normalization for Image Pyramids with Ef\ufb01cient Resampling (SNIPER), which adaptively\nsamples chips from multiple scales of an image pyramid, conditioned on the image content. We\nsample positive chips conditioned on the ground-truth instances and negative chips based on proposals\ngenerated by a region proposal network. Under the same conditions (\ufb01xed batch normalization),\nwe show that SNIPER performs as well as the multi-scale strategy proposed in SNIP [33] while\nreducing the number of pixels processed by a factor of 3 during training on the COCO dataset. Since\nSNIPER is trained on 512x512 size chips, it can reap the bene\ufb01ts of a large batch size and training\nwith batch-normalization on a single GPU node. In particular, we can use a batch size of 20 per GPU\n(leading to a total batch size of 160), even with a ResNet-101 based Faster-RCNN detector. While\nbeing ef\ufb01cient, SNIPER obtains competitive performance on the COCO detection dataset even with\nsimple detection architectures like Faster-RCNN.\n\n2 Background\n\nDeep learning based object detection algorithms have primarily evolved from the R-CNN detector\n[11], which started with object proposals generated with an unsupervised algorithm [34], resized\nthese proposals to a canonical 224x224 size image and classi\ufb01ed them using a convolutional neural\nnetwork [19]. This model is scale invariant, but the computational cost for training and inference\nfor R-CNN scales linearly with the number of proposals. To alleviate this computational bottleneck,\nFast-RCNN [10] proposed to project region proposals to a high level convolutional feature map\nand use the pooled features as a semantic representation for region proposals. In this process, the\ncomputation is shared for the convolutional layers and only lightweight fully connected layers are\napplied on each proposal. However, convolution for objects of different sizes is performed at a single\nscale, which destroys the scale invariance properties of R-CNN. Hence, inference at multiple scales\nis performed and detections from multiple scales are combined by selecting features from a pair of\nadjacent scales closer to the resolution of the pre-trained network [13, 10]. The Fast-RCNN model\nhas since become the de-facto approach for classifying region proposals as it is fast and also captures\nmore context in its features, which is lacking in RCNN.\nIt is worth noting that in multi-scale training, Fast-RCNN upsamples and downsamples every\nproposal (whether small or big) in the image. This is unlike R-CNN, where each proposal is resized\nto a canonical size of 224x224 pixels. Large objects are not upsampled and small objects are not\ndownsampled in R-CNN. In this regard, R-CNN more appropriately does not up/downsample every\npixel in the image but only in those regions which are likely to contain objects to an appropriate\nresolution. However, R-CNN does not share the convolutional features for nearby proposals like\nFast-RCNN, which makes it slow. To this end, we propose SNIPER, which retains the bene\ufb01ts\nof both these approaches by generating scale speci\ufb01c context-regions (chips) that cover maximum\nproposals at a particular scale. SNIPER classi\ufb01es all the proposals inside a chip like Fast-RCNN\nwhich enables us to perform ef\ufb01cient classi\ufb01cation of multiple proposals within a chip. As SNIPER\ndoes not upsample the image where there are large objects and also does not process easy background\nregions, it is signi\ufb01cantly faster compared to a Fast-RCNN detector trained on an image pyramid.\nSNIP [33] is also trained on almost all the pixels of the image pyramid (like Fast-RCNN), although\ngradients arising from objects of extreme resolutions are ignored. In particular, 2 resolutions of the\nimage pyramid (480 and 800 pixels) always engage in training and multiple 1000 pixel crops are\nsampled out of the 1400 pixel resolution of the image in the \ufb01nest scale. SNIPER takes this cropping\nprocedure to an extreme level by sampling 512 pixels crops from 3 scales of an image pyramid. At\nextreme scales (like 3x), SNIPER observes less than one tenth of the original content present in the\nimage! Unfortunately, as SNIPER chips generated only using ground-truth instances are very small\ncompared to the resolution of the original image, a signi\ufb01cant portion of the background does not\nparticipate in training. This causes the false positive rate to increase. Therefore, it is important to\n\n2\n\n\fFigure 1: SNIPER Positive chip selection . SNIPER adaptively samples context regions (aka chips)\nbased on the presence of objects inside the image. Left side: The image, ground-truth boxes\n(represented by green lines), and the chips in the original image scale (represented by the blue, yellow,\npink, and purple rectangles). Right side: Down/up-sampling is performed considering the size of the\nobjects. Covered objects are shown in green and invalid objects in the corresponding scale are shown\nas red rectangles.\n\ngenerate chips for background regions as well. In SNIPER, this is achieved by randomly sampling\na \ufb01xed number of chips (maximum of 2 in this paper) from regions which are likely to cover false\npositives. To \ufb01nd such regions, we train a lightweight RPN network with a short schedule. The\nproposals of this network are used to generate chips for regions which are likely to contain false\npositives (this could potentially be replaced with unsupervised proposals like EdgeBoxes [42] as\nwell). After adding negative chip sampling, the performance of SNIPER matches SNIP, but it is 3\ntimes faster! Since we are able to obtain similar performance by observing less than one tenth of the\nimage, it implies that very large context during training is not important for training high-performance\ndetectors but sampling regions containing hard negatives is.\n\n3 SNIPER\n\nWe describe the major components of SNIPER in this section. One is positive/negative chip mining\nand the other is label assignment after chips are generated. Finally, we will discuss the bene\ufb01ts of\ntraining with SNIPER.\n\n3.1 Chip Generation\nSNIPER generates chips Ci at multiple scales {s1, s2, .., si, ..sn} in the image. For each scale, the\nimage is \ufb01rst re-sized to width (Wi) and height (Hi). On this canvas, K \u00d7 K pixel chips are placed\nat equal intervals of d pixels (we set d to 32 in this paper). This leads to a two-dimensional array of\nchips at each scale.\n\nmin, ri\n\n3.2 Positive Chip Selection\nFor each scale, there is a desired area range Ri = [ri\nmax], i \u2208 [1, n] which determines which\nground-truth boxes/proposals participate in training for each scale i. The valid list of ground-truth\nbounding boxes which lie in Ri are referred to as Gi. Then, chips are greedily selected so that\nmaximum number of valid ground-truth boxes (Gi) are covered. A ground-truth box is said to be\ncovered if it is completely enclosed inside a chip. All the positive chips from a scale are combined\nper image and are referred to as Ci\npos. For each ground-truth bounding box, there always exists a chip\nwhich covers it. Since consecutive Ri contain overlapping intervals, a ground-truth bounding box\nmay be assigned to multiple chips at different scales. It is also possible that the same ground-truth\nbounding box may be in multiple chips from the same scale. Ground-truth instances which have\na partial overlap (IoU > 0) with a chip are cropped. All the cropped ground-truth boxes (valid or\ninvalid) are retained in the chip and are used in label assignment.\n\n3\n\nCoarsest ScaleIntermediate ScaleFinest ScaleFinest Scale\fFigure 2: SNIPER negative chip selection. First row: the image and the ground-truth boxes. Bottom\nrow: negative proposals not covered in positive chips (represented by red circles located at the center\nof each proposal for the clarity) and the generated negative chips based on the proposals (represented\nby orange rectangles).\n\nIn this way, every ground-truth box is covered at the appropriate scale. Since the crop-size is much\nsmaller than the resolution of the image (i.e. more than 10x smaller for high-resolution images),\nSNIPER does not process most of the background at high-resolutions. This leads to signi\ufb01cant\nsavings in computation and memory requirement while processing high-resolution images. We\nillustrate this with an example shown in Figure 1. The left side of the \ufb01gure shows the image with the\nground-truth boxes represented by green bounding boxes. Other colored rectangles on the left side\nof the \ufb01gure show the chips generated by SNIPER in the original image resolution which cover all\nobjects. These chips are illustrated on the right side of the \ufb01gure with the same border color. Green\nand red bounding boxes represent the valid and invalid ground-truth objects corresponding to the\nscale of the chip. As can be seen, in this example, SNIPER ef\ufb01ciently processes all ground-truth\nobjects in an appropriate scale by forming 4 low-resolution chips.\n\n3.3 Negative Chip Selection\n\nAlthough positive chips cover all the positive instances, a signi\ufb01cant portion of the background is\nnot covered by them. Incorrectly classifying background increases the false positive rate. In current\nobject detection algorithms, when multi-scale training is performed, every pixel in the image is\nprocessed at all scales. Although training on all scales reduces the false positive rate, it also increases\ncomputation. We posit that a signi\ufb01cant amount of the background is easy to classify and hence, we\ncan avoid performing any computation in those regions. So, how do we eliminate regions which are\neasy to classify? A simple approach is to employ object proposals to identify regions where objects\nare likely to be present. After all, our classi\ufb01er operates on region proposals and if there are no region\nproposals in a part of the image, it implies that it is very easy to classify as background. Hence, we\ncan ignore those parts of the image during training.\nTo this end, for negative chip mining, we \ufb01rst train RPN for a couple of epochs. No negative chips\nare used for training this network. The task of this network is to roughly guide us in selecting regions\nwhich are likely to contain false positives, so it is not necessary for it to be very accurate. This RPN is\nused to generate proposals over the entire training set. We assume that if no proposals are generated\nin a major portion of the image by RPN, then it is unlikely to contain an object instance. For negative\nchip selection, for each scale i, we \ufb01rst remove all the proposals which have been covered in Ci\npos.\nThen, for each scale i, we greedily select all the chips which cover at least M proposals in Ri. This\ngenerates a set of negative chips for each scale per image, Ci\nneg. During training, we randomly sample\na \ufb01xed number of negative chips per epoch (per image) from this pool of negative chips which are\ni=1 Ci\nneg. Figure 2 shows examples of the generated negative chips\nby SNIPER. The \ufb01rst row shows the image and the ground-truth boxes. In the bottom row, we show\nthe proposals not covered by Ci\npos and the corresponding negative chips generated (the orange boxes).\nHowever, for clarity, we represent each proposal by a red circle in its center. As illustrated, SNIPER\nonly processes regions which likely contain false positives, leading to faster processing time.\n\ngenerated from all scales, i.e.(cid:83)n\n\n4\n\n\f3.4 Label Assignment\n\nOur network is trained end to end on these chips like Faster-RCNN, i.e. it learns to generate proposals\nas well as classify them with a single network. While training, proposals generated by RPN are\nassigned labels and bounding box targets (for regression) based on all the ground-truth boxes which\nare present inside the chip. We do not \ufb01lter ground-truth boxes based on Ri. Instead, proposals\nwhich do not fall in Ri are ignored during training. So, a large ground-truth box which is cropped,\ncould generate a valid proposal which is small. Like Fast-RCNN, we mark any proposal which has\nan overlap greater than 0.5 with a ground-truth box as positive and assign bounding-box targets for\nthe proposal. Our network is trained end to end and we generate 300 proposals per chip. We do not\napply any constraint that a fraction of these proposals should be re-sampled as positives [29], as in\nFast-RCNN. We did not use OHEM [32] for classi\ufb01cation and use a simple softmax cross-entropy\nloss for classi\ufb01cation. For assigning RPN labels, we use valid ground-truth boxes to assign labels and\ninvalid ground-truth boxes to invalidate anchors, as done in SNIP [33].\n\n3.5 Bene\ufb01ts\n\nFor training, we randomly sample chips from the whole dataset for generating a batch. On average,\nwe generate \u223c 5 chips of size 512x512 per image on the COCO dataset (including negative chips)\nwhen training on three scales (512/ms 2, 1.667, 3). This is only 30% more than the number of pixels\nprocessed per image when single scale training is performed with an image resolution of 800x1333.\nSince all our images are of the same size, data is much better packed leading to better GPU utilization\nwhich easily overcomes the extra 30% overhead. But more importantly, we reap the bene\ufb01ts of\nmulti-scale training on 3 scales, large batch size and training with batch-normalization without any\nslowdown in performance on a single 8 GPU node!.\nIt is commonly believed that high resolution images (e.g. 800x1333) are necessary for instance\nlevel recognition tasks. Therefore, for instance level recognition tasks, it was not possible to train\nwith batch-normalization statistics computed on a single GPU. Methods like synchronized batch-\nnormalization [22, 41] or training on 128 GPUs [28] have been proposed to alleviate this problem.\nSynchronized batch-normalization slows down training signi\ufb01cantly and training on 128 GPUs is\nalso impractical for most people. Therefore, group normalization [35] has been recently proposed so\nthat instance level recognition tasks can bene\ufb01t from another form of normalization in a low batch\nsetting during training. With SNIPER, we show that the image resolution bottleneck can be alleviated\nfor instance level recognition tasks. As long as we can cover negatives and use appropriate scale\nnormalization methods, we can train with a large batch size of resampled low resolution chips, even\non challenging datasets like COCO. Our results suggest that context beyond a certain \ufb01eld of view\nmay not be bene\ufb01cial during training. It is also possible that the effective receptive \ufb01eld of deep\nneural networks is not large enough to leverage far away pixels in the image, as suggested in [24].\nIn very large datasets like OpenImagesV4 [18] containing 1.7 million images, most objects are large\nand images provided are high resolution (1024x768), so it is less important to upsample images\nby 3\u00d7. In this case, with SNIPER, we generate 3.5 million chips of size 512x512 using scales of\n(512/ms, 1). Note that SNIPER also performs adaptive downsampling. Since the scales are smaller,\nchips would cover more background, due to which the impact of negative sampling is diminished. In\nthis case (of positive chip selection), SNIPER processes only half the number of pixels compared to\nna\u00efve multi-scale training on the above mentioned scales in OpenImagesV4. Due to this, we were\nable to train Faster-RCNN with a ResNet-101 backbone on 1.7 million images in just 3 days on a\nsingle 8 GPU node!\n\n4 Experimental Details\n\nWe evaluate SNIPER on the COCO dataset for object detection. COCO contains 123,000 images\nin the training and validation set and 20,288 images in the test-dev set. We train on the combined\ntraining and validation set and report results on the test-dev set. Since recall for proposals is not\nprovided by the evaluation server, we train on 118,000 images and report recall on the remaining\n5,000 images (commonly referred to as the minival set).\n\n2max(widthim,heightim)\n\n5\n\n\fMethod\n\nResNet-101 With Neg\nResNet-101 W/o Neg\n\nAR AR50 AR75\n76.9\n65.4\n65.4\n77.6\n\n93.2\n93.2\n\n0-25\n41.3\n40.8\n\n25-50\n65.8\n65.7\n\n50-100\n74.5\n74.7\n\n100-200\n\n200-300\n\n76.9\n77.4\n\n78.7\n79.3\n\nTable 1: We plot the recall for SNIPER with and without negatives. Surprisingly, recall is not effected\nby negative chip sampling\n\nOn COCO, we train SNIPER with a batch-size of 128 and with a learning rate of 0.015. We use a\nchip size of 512\u00d7 512 pixels. Training scales are set to (512/ms, 1.667, 3) where ms is the maximum\nvalue width and height of the image3. The desired area ranges (i.e. Ri) are set to (0,802), (322, 1502),\nand (1202, inf) for each of the scales respectively. Training is performed for a total of 6 epochs with\nstep-down at the end of epoch 5. Image \ufb02ipping is used as a data-augmentation technique. Every\nepoch requires 11,000 iterations. For training RPN without negatives, each epoch requires 7000\niterations. We use RPN for generating negative chips and train it for 2 epochs with a \ufb01xed learning\nrate of 0.015 without any step-down. Therefore, training RPN for 2 epochs requires less than 20% of\nthe total training time. RPN proposals are extracted from all scales. Note that inference takes 1/3 the\ntime for a full forward-backward pass and we do not perform any \ufb02ipping for extracting proposals.\nHence, this process is also ef\ufb01cient. We use mixed precision training as described in [27]. To this\nend, we re-scale weight-decay by 100, drop the learning rate by 100 and rescale gradients by 100.\nThis ensures that we can train with activations of half precision (and hence \u223c 2x larger batch size)\nwithout any loss in accuracy. We use fp32 weights for the \ufb01rst convolution layer, last convolution\nlayer in RPN (classi\ufb01cation and regression) and the fully connected layers in Faster-RCNN.\nWe evaluate SNIPER using a popular detector, Faster-RCNN with ResNets [14, 15] and MobileNetV2\n[31]. Proposals are generated using RPN on top of conv4 features and classi\ufb01cation is performed after\nconcatenating conv4 and conv5 features. In the conv5 branch, we use deformable convolutions and\na stride of 1. We use a 512 dimensional feature map in RPN. For the classi\ufb01cation branch, we \ufb01rst\nproject the concatenated feature map to 256 dimensions and then add 2 fully connected layers with\n1024 hidden units. For lightweight networks like MobileNetv2 [31], to preserve the computational\nprocessing power of the network, we did not make any architectural changes to the network like\nchanging the stride of the network or added deformable convolutions. We reduced the RPN dimension\nto 256 and size of fc layers to 512 from 1024. RPN and classi\ufb01cation branch are both applied on the\nlayer with stride 32 for MobileNetv2.\nSNIPER generates 1.2 million chips for the COCO dataset after the images are \ufb02ipped. This results\nin around 5 chips per image. In some images which contain many object instances, SNIPER can\ngenerate as many as 10 chips and others where there is a single large salient object, it would only\ngenerate a single chip. In a sense, it reduces the imbalance in gradients propagated to an instance\nlevel which is present in detectors which are trained on full resolution images. At least in theory,\ntraining on full resolution images is biased towards large object instances.\n\n4.1 Recall Analysis\n\nWe observe that recall (averaged over multiple overlap thresholds 0.5:0.05:0.95) for RPN does not\ndecrease if we do not perform negative sampling. This is because recall does not account for false\npositives. As shown in Section 4.2, this is in contrast to mAP for detection in which negative sampling\nplays an important role. Moreover, in positive chip sampling, we do cover every ground truth sample.\nTherefore, for generating proposals, it is suf\ufb01cient to train on just positive samples. This result further\nbolsters SNIPER\u2019s strategy of \ufb01nding negatives based on an RPN in which the training is performed\njust on positive samples.\n\n4.2 Negative Chip Mining and Scale\n\nSNIPER uses negative chip mining to reduce the false positive rate while speeding up the training by\nskipping the easy regions inside the image. As proposed in Section 3.3, we use a region proposal\nnetwork trained with a short learning schedule to \ufb01nd such regions. To evaluate the effectiveness of\nour negative mining approach, we compare SNIPER\u2019s mean average precision with a slight variant\n\n3For the \ufb01rst scale, zero-padding is used if the smaller side of the image becomes less than 512 pixels.\n\n6\n\n\fMethod\nSNIPER\n\nBackbone\nResNet-101\nResNet-101\nSNIPER w/o negatives ResNet-101\n\nSNIPER 2 scale\n\nAP\n46.1\n43.3\n43.4\n\nAP50 AP75 APS APM APL\n58.1\n67.0\n56.1\n63.7\n62.8\n56.2\n\n48.9\n44.7\n45.2\n\n51.6\n48.6\n48.8\n\n29.6\n27.1\n27.4\n\nTable 2: The effect training on 2 scales (1.667 and max size of 512). We also show the impact in\nperformance when no negative mining is performed.\n\nwhich only uses positive chips during training (denoted as SNIPER w/o neg). All other parameters\nremain the same. Table 2 compares the performance of these models. The proposed negative chip\nmining approach noticeably improves AP for all localization thresholds and object sizes. Noticeably,\nnegative chip mining improves the average precision from 43.4 to 46.1. This is in contrast to the last\nsection where we were evaluating proposals. This is because mAP is affected by false positives. If we\ndo not include regions in the image containing negatives which are similar in appearance to positive\ninstances, it would increase our false positive rate and adversely affect detection performance.\nSNIPER is an ef\ufb01cient multi-scale training algorithm. In all experiments in this paper we use the\naforementioned three scales (See Section 4 for the details). To show that SNIPER effectively bene\ufb01ts\nfrom multi-scale training, we reduce the number of scales from 3 to 2 by dropping the high resolution\nscale. Table 2 shows the mean average precision for SNIPER under these two settings. As can be\nseen, by reducing the number of scales, the performance consistently drops by a large margin on all\nevaluation metrics.\n\n4.3 Timing\n\nIt takes 14 hours to train SNIPER end to end on a 8 GPU V100 node with a Faster-RCNN detector\nwhich has a ResNet-101 backbone. It is worth noting that we train on 3 scales of an image pyramid\n(max size of 512, 1.667 and 3). Training RPN is much more ef\ufb01cient and it only takes 2 hours for\npre-training. Not only is SNIPER ef\ufb01cient in training, it can also process around 5 images per second\non a single V100 GPU. For better utilization of resources, we run multiple processes in parallel\nduring inference and compute the average time it takes to process a batch of 100 images.\n\n4.4\n\nInference\n\nWe perform inference on an image pyramid and scale the original image to the following resolutions\n(480, 512), (800, 1280) and (1400, 2000). The \ufb01rst element is the minimum size with the condition\nthat the maximum size does not exceed the second element. The valid ranges for training and\ninference are similar to SNIP [33]. For combining the detections, we use Soft-NMS [4]. We do not\nperform \ufb02ipping [39], iterative bounding box regression [9] or mask tightening [22].\n\n4.5 Comparison with State-of-the-art\n\nIt is dif\ufb01cult to fairly compare different detectors as they differ in backbone architectures (like ResNet\n[14], ResNext [36], Xception [6]), pre-training data (e.g. ImageNet-5k, JFT [16], OpenImages\n[18]), different structures in the underlying network (e.g multi-scale features [20, 26], deformable\nconvolutions [8], heavier heads [28], anchor sizes, path aggregation [22]), test time augmentations\nlike \ufb02ipping, mask tightening, iterative bounding box regression etc.\nTherefore, we compare our results with SNIP [33], which is a recent method for training object detec-\ntors on an image pyramid. The results are presented in Table 3. Without using batch normalization\n[17], SNIPER achieves comparable results. While SNIP [33] processes almost all the image pyramid,\nSNIPER on the other hand, reduces the computational cost by skipping easy regions. Moreover,\nsince SNIPER operates on a lower resolution input, it reduces the memory footprint. This allows\nus to increase the batch size and unlike SNIP [33], we can bene\ufb01t from batch normalization during\ntraining. With batch normalization, SNIPER signi\ufb01cantly outperforms SNIP in all metrics. It should\nbe noted that not only the proposed method is more accurate, it is also 3\u00d7 faster during training.\nTo the best of our knowledge, for a Faster-RCNN architecture with a ResNet-101 backbone (with\ndeformable convolutions), our reported result of 46.1% is state-of-the-art. This result improves to\n\n7\n\n\fMethod\n\nSSD\nSNIP\n\nSNIPER\n\nBackbone\n\nMobileNet-v2\n\nResNet-50 (\ufb01xed BN)\nResNet-101 (\ufb01xed BN)\n\nMobileNet-V2\n\nResNet-50 (\ufb01xed BN)\n\nResNet-101\n\nResNet-101 + OpenImages\n\nResNet-101 + OpenImages + Seg Binary\nResNet-101 + OpenImages + Seg Softmax\nSNIPER ResNet-101 + OpenImages + Seg Softmax\nResNet-101 + OpenImages + Seg Binary\nSNIPER\n\nAP50 AP75 APS APM APL\n\nAP\n22.1\n43.6\n44.4\n34.1\n43.5\n46.1\n46.8\n47.1\n47.6\n38.9\n41.3\n\n-\n\n65.2\n66.2\n54.4\n65.0\n67.0\n67.4\n67.8\n68.5\n62.9\n65.4\n\n-\n\n48.8\n49.9\n37.7\n48.6\n51.6\n52.5\n52.8\n53.4\n41.8\n44.9\n\n-\n\n26.4\n27.3\n18.2\n26.1\n29.6\n30.5\n30.2\n30.9\n19.6\n21.4\n\n-\n\n46.5\n47.4\n36.9\n46.3\n48.9\n49.4\n49.9\n50.6\n41.2\n43.5\n\n-\n\n55.8\n56.9\n46.2\n56.0\n58.1\n59.6\n60.2\n60.7\n55.0\n58.7\n\nTable 3: Ablation analysis and comparison with full resolution training. Last two rows show instance\nsegmentation results when the mask head is trained with N+1 way softmax loss and binary softmax\nloss for N classes.\n\n46.8% if we pre-train the detector on the OpenImagesV4 dataset. Adding an instance segmentation\nhead and training the detection network along with it improves the performance to 47.6%.\nWith our ef\ufb01cient batch inference pipeline, we can process 5 images per second on a single V100\nGPU and still obtain an mAP of 47.6%. This implies that on modern GPUs, it is practical to perform\ninference on an image pyramid which includes high resolutions like 1400x2000. We also show results\nfor Faster-RCNN trained with MobileNetV2. It obtains an mAP of 34.1% compared to the SSDLite\n[31] version which obtained 22.1%. This again highlights the importance of image pyramids (and\nSNIPER training) as we can improve the performance of the detector by 12%.\nWe also show results for instance segmentation. The network architecture is same as Mask-RCNN\n[12], just that we do not use FPN [20] and use the same detection architecture which was described\nfor object detection. For multi-tasking, we tried two variants of loss functions for training the mask\nbranch. One was a foreground-background softmax function for N classes and another was a N+1\nway softmax function. For instance segmentation, the network which is trained with 2-way Softmax\nloss for each class clearly performs better. But, for object detection, the N+1 way Softmax loss\nleads to slightly better results. We only use 3 scales during inference and do not perform \ufb02ipping,\nmask tightening, iterative bounding-box regression, padding masks before resizing etc. Our instance\nsegmentation results are preliminary and we have only trained 2 models so far.\n\n5 Related Work\n\nSNIPER bene\ufb01ts from multiple techniques which were developed over the last year. Notably, it was\nshown that it is important to train with batch normalization statistics [28, 22, 41] for tasks like object\ndetection and semantic segmentation. This is one important reason for SNIPER\u2019s better performance.\nSNIPER also bene\ufb01ts from a large batch size which was shown to be effective for object detection\n[28]. Like SNIP [33], SNIPER ignores gradients of objects at extreme scales in the image pyramid to\nimprove multi-scale training.\nIn the past, many different methods have been proposed to understand the role of context [38, 1, 25],\nscale [5, 37, 20, 26] and sampling [21, 32, 2, 3]. Considerable importance has been given to\nleveraging features of different layers of the network and designing architectures for explicitly\nencoding context/multi-scale information [26, 23, 39, 40] for classi\ufb01cation. Our results highlight that\ncontext may not be very important for training high performance object detectors.\n\n6 Conclusion and Future Work\n\nWe presented an algorithm for ef\ufb01cient multi-scale training which sampled low resolution chips from\na multi-scale image pyramid to accelerate multi-scale training by a factor of 3 times. In doing so,\nSNIPER did not compromise on the performance of the detector due to effective sampling techniques\nfor positive and negative chips. As SNIPER operates on re-sampled low resolution chips, it can be\ntrained with a large batch size on a single GPU which brings it closer to the protocol for training image\n\n8\n\n\fclassi\ufb01cation. This is in contrast with the common practice of training on high resolution images\nfor instance-level recognition tasks. In future, we would like to accelerate multi-scale inference,\nbecause a signi\ufb01cant portion of the background can be eliminated without performing expensive\ncomputation. It would also be interesting to evaluate at what chip resolution does context start to hurt\nthe performance of object detectors.\n\n7 Acknowledgement\n\nThe authors would like to thank an Amazon Machine Learning gift for the AWS credits used for\nthis research. The research is based upon work supported by the Of\ufb01ce of the Director of National\nIntelligence (ODNI), Intelligence Advanced Research Projects Activity (IARPA), via DOI/IBC\nContract Numbers D17PC00287 and D17PC00345. The U.S. Government is authorized to reproduce\nand distribute reprints for Governmental purposes not withstanding any copyright annotation thereon.\nDisclaimer: The views and conclusions contained herein are those of the authors and should not\nbe interpreted as necessarily representing the of\ufb01cial policies or endorsements, either expressed or\nimplied of IARPA, DOI/IBC or the U.S. Government.\n\nReferences\n[1] S. Bell, C. Lawrence Zitnick, K. Bala, and R. Girshick. Inside-outside net: Detecting objects in context\nwith skip pooling and recurrent neural networks. In Proceedings of the IEEE Conference on Computer\nVision and Pattern Recognition, pages 2874\u20132883, 2016.\n\n[2] V. P. Boda and P. Narayan. Sampling rate distortion. IEEE Transactions on Information Theory, 63(1):563\u2013\n\n574, 2017.\n\n[3] V. P. Boda and P. Narayan. Universal sampling rate distortion. IEEE Transactions on Information Theory,\n\n2018.\n\n[4] N. Bodla, B. Singh, R. Chellappa, and L. S. Davis. Soft-nms\u2014improving object detection with one line of\ncode. In 2017 IEEE International Conference on Computer Vision (ICCV), pages 5562\u20135570. IEEE, 2017.\n\n[5] Z. Cai, Q. Fan, R. S. Feris, and N. Vasconcelos. A uni\ufb01ed multi-scale deep convolutional neural network\n\nfor fast object detection. In European Conference on Computer Vision, pages 354\u2013370. Springer, 2016.\n\n[6] F. Chollet. Xception: Deep learning with depthwise separable convolutions. CVPR, 2017.\n\n[7] M. Corbetta and G. L. Shulman. Control of goal-directed and stimulus-driven attention in the brain. Nature\n\nreviews neuroscience, 3(3):201, 2002.\n\n[8] J. Dai, H. Qi, Y. Xiong, Y. Li, G. Zhang, H. Hu, and Y. Wei. Deformable convolutional networks. ICCV,\n\n2017.\n\n[9] S. Gidaris and N. Komodakis. Locnet: Improving localization accuracy for object detection. In Proceedings\n\nof the IEEE Conference on Computer Vision and Pattern Recognition, pages 789\u2013798, 2016.\n\n[10] R. Girshick. Fast r-cnn. In Computer Vision (ICCV), 2015 IEEE International Conference on, pages\n\n1440\u20131448. IEEE, 2015.\n\n[11] R. Girshick, J. Donahue, T. Darrell, and J. Malik. Rich feature hierarchies for accurate object detection and\nsemantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition,\npages 580\u2013587, 2014.\n\n[12] K. He, G. Gkioxari, P. Doll\u00e1r, and R. Girshick. Mask r-cnn. In Computer Vision (ICCV), 2017 IEEE\n\nInternational Conference on, pages 2980\u20132988. IEEE, 2017.\n\n[13] K. He, X. Zhang, S. Ren, and J. Sun. Spatial pyramid pooling in deep convolutional networks for visual\n\nrecognition. In European Conference on Computer Vision, pages 346\u2013361. Springer, 2014.\n\n[14] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the\n\nIEEE conference on computer vision and pattern recognition, pages 770\u2013778, 2016.\n\n[15] K. He, X. Zhang, S. Ren, and J. Sun. Identity mappings in deep residual networks. In European Conference\n\non Computer Vision, pages 630\u2013645. Springer, 2016.\n\n9\n\n\f[16] G. Hinton, O. Vinyals, and J. Dean. Distilling the knowledge in a neural network. arXiv preprint\n\narXiv:1503.02531, 2015.\n\n[17] S. Ioffe and C. Szegedy. Batch normalization: Accelerating deep network training by reducing internal\n\ncovariate shift. ICML, 2015.\n\n[18] I. Krasin, T. Duerig, N. Alldrin, V. Ferrari, S. Abu-El-Haija, A. Kuznetsova, H. Rom, J. Ui-\njlings, S. Popov, S. Kamali, M. Malloci, J. Pont-Tuset, A. Veit, S. Belongie, V. Gomes, A. Gupta,\nC. Sun, G. Chechik, D. Cai, Z. Feng, D. Narayanan, and K. Murphy. Openimages: A pub-\nlic dataset for large-scale multi-label and multi-class image classi\ufb01cation. Dataset available from\nhttps://storage.googleapis.com/openimages/web/index.html, 2017.\n\n[19] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition.\n\nProceedings of the IEEE, 86(11):2278\u20132324, 1998.\n\n[20] T.-Y. Lin, P. Doll\u00e1r, R. Girshick, K. He, B. Hariharan, and S. Belongie. Feature pyramid networks for\n\nobject detection. In CVPR, pages 936\u2013944, 2017.\n\n[21] T.-Y. Lin, P. Goyal, R. Girshick, K. He, and P. Doll\u00e1r. Focal loss for dense object detection. IEEE\n\ntransactions on pattern analysis and machine intelligence, 2018.\n\n[22] S. Liu, L. Qi, H. Qin, J. Shi, and J. Jia. Path aggregation network for instance segmentation. CVPR, 2018.\n\n[23] W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C.-Y. Fu, and A. C. Berg. Ssd: Single shot multibox\n\ndetector. In European conference on computer vision, pages 21\u201337. Springer, 2016.\n\n[24] W. Luo, Y. Li, R. Urtasun, and R. Zemel. Understanding the effective receptive \ufb01eld in deep convolutional\n\nneural networks. In Advances in Neural Information Processing Systems, pages 4898\u20134906, 2016.\n\n[25] R. Mottaghi, X. Chen, X. Liu, N.-G. Cho, S.-W. Lee, S. Fidler, R. Urtasun, and A. Yuille. The role of\ncontext for object detection and semantic segmentation in the wild. In Proceedings of the IEEE Conference\non Computer Vision and Pattern Recognition, pages 891\u2013898, 2014.\n\n[26] M. Najibi, P. Samangouei, R. Chellappa, and L. Davis. SSH: Single stage headless face detector. In\nProceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4875\u20134884,\n2017.\n\n[27] S. Narang, G. Diamos, E. Elsen, P. Micikevicius, J. Alben, D. Garcia, B. Ginsburg, M. Houston,\n\nO. Kuchaiev, G. Venkatesh, et al. Mixed precision training. ICLR, 2018.\n\n[28] C. Peng, T. Xiao, Z. Li, Y. Jiang, X. Zhang, K. Jia, G. Yu, and J. Sun. Megdet: A large mini-batch object\n\ndetector. CVPR, 2018.\n\n[29] S. Ren, K. He, R. Girshick, and J. Sun. Faster r-cnn: Towards real-time object detection with region\n\nproposal networks. In Advances in neural information processing systems, pages 91\u201399, 2015.\n\n[30] R. A. Rensink. The dynamic representation of scenes. Visual cognition, 7(1-3):17\u201342, 2000.\n\n[31] M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L.-C. Chen. Mobilenetv2: Inverted residuals and\nlinear bottlenecks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,\npages 4510\u20134520, 2018.\n\n[32] A. Shrivastava, A. Gupta, and R. Girshick. Training region-based object detectors with online hard example\nmining. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages\n761\u2013769, 2016.\n\n[33] B. Singh and L. S. Davis. An analysis of scale invariance in object detection-snip. CVPR, 2018.\n\n[34] J. R. Uijlings, K. E. Van De Sande, T. Gevers, and A. W. Smeulders. Selective search for object recognition.\n\nInternational journal of computer vision, 104(2):154\u2013171, 2013.\n\n[35] Y. Wu and K. He. Group normalization. arXiv:1803.08494, 2018.\n\n[36] S. Xie, R. Girshick, P. Doll\u00e1r, Z. Tu, and K. He. Aggregated residual transformations for deep neural\nIn Computer Vision and Pattern Recognition (CVPR), 2017 IEEE Conference on, pages\n\nnetworks.\n5987\u20135995. IEEE, 2017.\n\n[37] F. Yang, W. Choi, and Y. Lin. Exploit all the layers: Fast and accurate cnn object detector with scale\ndependent pooling and cascaded rejection classi\ufb01ers. In Proceedings of the IEEE Conference on Computer\nVision and Pattern Recognition, pages 2129\u20132137, 2016.\n\n10\n\n\f[38] F. Yu and V. Koltun. Multi-scale context aggregation by dilated convolutions.\n\narXiv:1511.07122, 2015.\n\narXiv preprint\n\n[39] S. Zagoruyko, A. Lerer, T.-Y. Lin, P. O. Pinheiro, S. Gross, S. Chintala, and P. Doll\u00e1r. A multipath network\n\nfor object detection. BMVC, 2016.\n\n[40] X. Zeng, W. Ouyang, J. Yan, H. Li, T. Xiao, K. Wang, Y. Liu, Y. Zhou, B. Yang, Z. Wang, et al. Crafting\n\ngbd-net for object detection. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017.\n\n[41] H. Zhao, J. Shi, X. Qi, X. Wang, and J. Jia. Pyramid scene parsing network. In IEEE Conf. on Computer\n\nVision and Pattern Recognition (CVPR), pages 2881\u20132890, 2017.\n\n[42] C. L. Zitnick and P. Doll\u00e1r. Edge boxes: Locating object proposals from edges. In European Conference\n\non Computer Vision, pages 391\u2013405. Springer, 2014.\n\n11\n\n\f", "award": [], "sourceid": 5679, "authors": [{"given_name": "Bharat", "family_name": "Singh", "institution": "University of Maryland, College Park"}, {"given_name": "Mahyar", "family_name": "Najibi", "institution": "University of Maryland"}, {"given_name": "Larry", "family_name": "Davis", "institution": "University of Maryland"}]}