{"title": "Consistency-based Semi-supervised Learning for Object detection", "book": "Advances in Neural Information Processing Systems", "page_first": 10759, "page_last": 10768, "abstract": "Making a precise annotation in a large dataset is crucial to the performance of object detection. While the object detection task requires a huge number of annotated samples to guarantee its performance, placing bounding boxes for every object in each sample is time-consuming and costs a lot. To alleviate this problem, we propose a Consistency-based Semi-supervised learning method for object Detection (CSD), which is a way of using consistency constraints as a tool for enhancing detection performance by making full use of available unlabeled data. Specifically, the consistency constraint is applied not only for object classification but also for the localization. We also proposed Background Elimination (BE) to avoid the negative effect of the predominant backgrounds on the detection performance. We have evaluated the proposed CSD both in single-stage and two-stage detectors and the results show the effectiveness of our method.", "full_text": "Consistency-based Semi-supervised Learning for\n\nObject Detection\n\nJisoo Jeong\u2217, Seungeui Lee\u2217, Jeesoo Kim & Nojun Kwak\n\nDepartment of Transdisciplinary Studies\n\nGraduate School of Convergence Science and Technology\n\nSeoul National University\n\nSeoul, Korea\n\n{soo3553, seungeui.lee, kimjiss0305, nojunk}@snu.ac.kr\n\nAbstract\n\nMaking a precise annotation in a large dataset is crucial to the performance of object\ndetection. While the object detection task requires a huge number of annotated\nsamples to guarantee its performance, placing bounding boxes for every object\nin each sample is time-consuming and costs a lot. To alleviate this problem, we\npropose a Consistency-based Semi-supervised learning method for object Detection\n(CSD), which is a way of using consistency constraints as a tool for enhancing\ndetection performance by making full use of available unlabeled data. Speci\ufb01cally,\nthe consistency constraint is applied not only for object classi\ufb01cation but also\nfor the localization. We also proposed Background Elimination (BE) to avoid the\nnegative effect of the predominant backgrounds on the detection performance. We\nhave evaluated the proposed CSD both in single-stage2 and two-stage detectors3\nand the results show the effectiveness of our method.\n\n1\n\nIntroduction\n\nLarge datasets with complete annotations are essential to the success of object detection. Training\nan object detection algorithm requires annotations in the level of bounding box as shown in Fig. 1\n(a). Labeling for object detection requires a pair of a category and a bounding box location for each\nobject within each image and it is known that it takes about 10 seconds for labeling an object [1, 2].\nAs such, labeling for object detection consumes enormous cost, time, and effort. For example, the\nCaltech pedestrian detection benchmark took about 400 hours to annotate 250k images [3].\nTo reduce the cost of such labeling, weakly supervised learning and Semi-Supervised Learning (SSL)\nmethods have been studied. Weakly supervised object detection methods [4, 5, 6, 7] are required to\nlearn only with image-level labeled data, as shown in Figure 1 (b). Although this takes less efforts than\nthe existing box-level labeling method, it results in a far inferior localization performance compared\nto fully supervised learning. The weakly semi-supervised object detection method [8, 9] uses fully\nlabeled data as well as weakly labeled data, as shown in Figure 1 (c). The complete semi-supervised\nobject detection method is to improve performance by using unlabeled data in combination with\nthe box-level labeled data, as shown in Figure 1 (d). Studies on complete semi-supervised object\ndetection have recently studied [10, 11] and we also deal with this problem.\nRecently, there have been studies that have improved the performance of semi-supervised learning\nusing self-training [10, 11]. They have improved the performance by utilizing high-con\ufb01dent samples\n\n\u2217Equal contribution\n2https://github.com/soo89/CSD-SSD\n3https://github.com/soo89/CSD-RFCN\n\n33rd Conference on Neural Information Processing Systems (NeurIPS 2019), Vancouver, Canada.\n\n\f(a) Supervised learning\n\n(b) Weakly supervised learning\n\n(c) Weakly semi-supervised learning\n\n(d) Semi-supervised learning\n\nFigure 1: Different types of object detection settings\n\nwith pseudo-labels in the training. However, these methods take a long time to train because they\npredict all unlabeled data in each iteration and learn the gradually increasing number of training\nsamples [12]. In addition, performances vary a lot depending on the number of pseudo-labeled\nsamples added.\nIn this paper, we propose Consistency-based Semi-supervised learning for object Detection (CSD)\nwhich is similar to the consistency regularization (CR) [13, 14, 15] that has shown state-of-the-art\nperformance in semi-supervised classi\ufb01cation [16]. CR helps train a model to be robust to given\nperturbed inputs. However, it is dif\ufb01cult to apply CR directly to the object detection problem where\nmultiple candidate boxes are generated for each image. Because images with different perturbation\nmay have different numbers of boxes with various locations and sizes, it is dif\ufb01cult to match boxes in\ngiven images. Therefore, we use the horizontally \ufb02ipped image so that one-to-one correspondence\nbetween the predicted boxes in the original and the \ufb02ipped images can be easily identi\ufb01ed. In our\nmethod, in addition to applying consistency constraint to the classi\ufb01cation results for each predicted\nbox, we propose a new consistency loss for \ufb01ne-tuning the location of the predicted box. Experimental\nresults show that each of these consistency losses can improve performance and we can get additional\nperformance improvement by combining these two.\nWe also observed that eliminating \u2018background\u2019 class bene\ufb01ts the proposed CSD, because the\npredominant \u2018background\u2019 class affects the consistency loss much. As a way of reducing the in\ufb02uence\nof the background and achieving improved performance, we propose the Background Elimination (BE)\nmethod which excludes boxes with high background probability in the computation of consistency\nloss.\nCSD can be applied to both the single-stage detector such as SSD [17] and the two-stage detector such\nas R-FCN [18]. Various ablation studies have been performed showing the bene\ufb01ts of the proposed\nconsistency losses for classi\ufb01cation and localization. Also the effect of BE has been experimentally\ncon\ufb01rmed. Experimental results show that the proposed CSD improves the detection performance for\nall the detectors experimented.\nOur main contributions can be summarized as follows:\n\u2022 We propose a novel consistency-based semi-supervised learning algorithm for object detection that\ncan be applied not only to single-stage detectors but also to two-stage detectors.\n\u2022 The proposed consistency constraints for object detection work well for both the classi\ufb01cation of a\nbounding box and the regression of its location.\n\u2022 We propose the BE method to mitigate the effect of background and show improvement of\nperformance in most cases.\n\n2\n\n\f2 Related Work\n\n2.1 Semi-Supervised Learning\nAn actual training environment usually provides a \ufb01nite number of labeled data (L = {(xl, yl)})\nand an unlimited number of unlabeled data (U = {(xu}). Many researches have tried to exploit the\npotential of unlabeled data since the majority of the real-world samples lack annotations. Generally,\nthere have been two methodologies to cope with these circumstances.\nSelf-training: Self-training methods train a model using labeled data and then make predictions on\nunlabeled data. If the top-1 prediction score for the input xu is greater than a threshold \u03c3, the pseudo\nlabel of xu is set as the class \u00afy whose score is the maximum.Then xu can be treated as a labeled data\nin the form of (xu, \u00afy) [19]. Repetitively applying this process can boost the model\u2019s performance\nbut impedes the whole training speed. In addition, depending on the threshold value \u03c3, the amount\nof added data varies a lot and this makes the performance unstable. A small number of additional\npseudo-labeled samples may not improve the performance enough while too many samples may harm\nthe performance with incorrect labeling.\nConsistency regularization: Consistency regularization applies perturbations to an input image x to\nobtain x(cid:48) and minimizes the difference between the outputs predictions f (x) and f (x(cid:48)) [13, 14, 15].\nIt does not require a label because the loss is determined by the difference between the outputs which\nis known to help smooth the manifold [16]. As mentioned above, this shows the state-of-the-art\nperformance in semi-supervised classi\ufb01cation problems.\n\n2.2 Semi-Supervised Learning for Object Detection\n\nObject detection: Object detection algorithms can be divided into two categories depending on\nwhether a region proposal network (RPN) is used or not. Algorithms that do not use RPN are\ncategorized as single-stage detectors, and the other algorithms are categorized as two-stage detectors.\nSingle-stage detectors perform classi\ufb01cation and localization in all the spatial location of feature\nmaps [17, 20]. There have been tremendous performance improvements using deep learning and\nthere are algorithms that are able to detect objects real-time on a desktop. Two-stage detectors are\nRPN-based algorithms, which detect object only for RoIs that have high possibility of containing an\nobject [18, 21].\nSemi-supervised learning for object detection: Until recently, most semi-supervised learning\nmethods for object detection are based on the self-training scheme. A representative method is the\nSelf-supervised Sample Mining (SSM) [10] algorithm, which improved performance by stitching high-\ncon\ufb01dence patches from unlabeled data to labeled data. SSM has used a method called \u2018evaluating\nconsistency\u2019 to make the pseudo box label robust. It operates as a mask to verify that the Intersection\nover Union (IoU) score between the previously detected box and the currently detected box is greater\nthan threshold \u03b3. Therefore, SSM differs from our method of directly using consistency losses. SSM\nrepeats the process of making an intermediate unlabeled data prediction and changing the training set.\nConsequently, SSM shares the same drawback as self-training.\n\n3 Method\n\nThe CSD to be presented works differently depending whether it is for a single-stage or for a two-stage\nobject detector. The overall CSD structure for single-stage and two-stage object detectors is depicted\nin Fig. 2 (a) and (b) respectively. The proposed structure is the combination of the \u03a0-model in SSL\n[13] and an object detection algorithm. To allow one-to-one correspondence of target objects, an\noriginal image, I, and its \ufb02ipped version, \u02c6I, are used as inputs. As in Fig. 2, a paired bounding box\nshould represent the same class and their localization information must remain consistent.\nDuring the training process, each mini-batch includes both labeled and unlabeled images. The labeled\nsamples are trained using the typical object detection approach. The consistency loss is additionally\napplied to both the labeled and unlabeled images. In section 3.1, we explain the association method of\ncorresponding boxes as well as the objective function used for training the object classi\ufb01er in a single\nstage object detector. Likewise, in section 3.2, we de\ufb01ne the objective function used for localization\n\n3\n\n\fFigure 2: Overall structure of our proposed method. (a) f k(I) and f k(cid:48)\n( \u02c6I) are extracted by a single\nstage detector from image I and \ufb02ipped image \u02c6I respectively. The supervised loss is computed\nbetween f k(I) and the ground truth for labeled data and the consistency loss is computed between\nf k(I) and f k(cid:48)\n( \u02c6I) for labeled and unlabeled data. (b) \u03c6(I) and \u03c6( \u02c6I) originate from the backbone\nnetwork and the RoI is computed only from \u03c6(I). \u02c6hk is obtained by \ufb02ipping hk to associate two\ncorresponding boxes and supervised and consistency losses are calculated in the same way as for the\nsingle stage detector.\n\nin both images. In the following sections afterward, we explain how these loss functions are utilized\nand show that our method is also applicable to a two-stage object detector.\n\n3.1 Consistency loss for classi\ufb01cation\n\ncls\n\ncls\n\nWe denote f p,r,c,d\n(I) as the output class probability vector after softmax operation corresponding to\nthe p-th pyramid, r-th row, c-th column and d-th default box. Since \u02c6I is a horizontally \ufb02ipped version\nof I, predictions of two images should be equivalent. Also we want to make these vectors, f p,r,c,d\n(I)\nand f p,r,c(cid:48),d\n( \u02c6I), share a very similar distribution where c(cid:48) = C \u2212 c + 1 and C is horizontal spatial\ndimension of the feature map. In semi-supervised learning, some candidates such as L2 distance or\nJensen-Shannon divergence (JSD) can be used as the consistency regularization loss. Among them,\nwe speci\ufb01cally take advantage of JSD for the following reasons. L2 loss treats all the classes equal\nand in our case, consistency loss for irrelevant classes with low probability can affect the classi\ufb01cation\nperformance much. We experimentally observed that the performance of SSL with L2 consistency\nloss is even worse than that of the supervised learning. To simplify the notation, we denote the\nlocation (p, r, c, d) as k and the horizontally opposite location (p, r, c(cid:48), d) as k(cid:48). The classi\ufb01cation\nconsistency loss used for a pair of bounding boxes in our method is given as below:\n\ncls\n\nlcon_cls(f k\n\ncls(I), f k(cid:48)\n\ncls( \u02c6I)) = JS(f k\n\ncls(I), f k(cid:48)\n\ncls( \u02c6I))\n\n(1)\n\nwhere JS represents the Jensen-Shannon Divergence. The overall consistency loss for classi\ufb01cation\nis then obtained from the average of loss values from all bounding box pairs:\n\nLcon\u2212c = Ek[lcon_cls(f k\n\ncls(I), f k(cid:48)\n\ncls( \u02c6I))]\n\n(2)\n\n3.2 Consistency loss for localization\n\nThe localization result for the k-th candidate box f k\nloc(I) consists of [\u2206cx, \u2206cy, \u2206w, \u2206h], which\nrepresent the displacement of the center and scale coef\ufb01cients of a candidate box, respectively. Unlike\nloc( \u02c6I) require a simple modi\ufb01cation to be equivalent to each\nthe pair (f k\n\nloc(I) and f k(cid:48)\n\ncls(I), f k(cid:48)\n\ncls( \u02c6I)), f k\n\n4\n\n\fother. Since the \ufb02ipping transformation makes \u2206 \u02c6cx move in the opposite direction, a negation should\nbe applied to correct it.\n\n\u2206 cxk \u21d0\u21d2 \u2212\u2206 \u02c6cxk(cid:48)\n\u2206 cyk, \u2206 wk, \u2206 hk \u21d0\u21d2 \u2206 \u02c6cyk(cid:48)\n\n, \u2206 \u02c6wk(cid:48)\n\n, \u2206 \u02c6hk(cid:48)\n\nThe localization consistency loss used for a single pair of bounding boxes in our method is given as\nbelow:\n\nlcon_loc(f k\n\nloc(I), f k(cid:48)\n\nloc( \u02c6I)) =\n\n((cid:107)\u2206cxk \u2212 (\u2212\u2206 \u02c6cxk(cid:48)\n1\n4\n+ (cid:107)\u2206wk \u2212 \u2206 \u02c6wk(cid:48)(cid:107)2+(cid:107)\u2206hk \u2212 \u2206\u02c6hk(cid:48)(cid:107)2)\n\n)(cid:107)2+(cid:107)\u2206cyk \u2212 \u2206 \u02c6cyk(cid:48)(cid:107)2\n\nThe localization loss of each pair of bounding boxes and the total consistency loss are computed in\nthe same principle as in the previous section:\n\nLcon\u2212l = Ek[lcon_loc(f k\n\nloc(I), f k(cid:48)\n\nloc( \u02c6I))]\n\n3.3 Overall loss for object detection\n\n(3)\n\n(4)\n\n(6)\n\n(7)\n\n(8)\n\nThe total consistency loss is composed of the losses from section 3.1 and 3.2 as in\n\nLcon = Lcon\u2212c + Lcon\u2212l\n\n(5)\nEventually, the \ufb01nal loss L is composed of the original object detector\u2019s classi\ufb01cation loss Lc and\nlocalization loss Ll, in addition to the consistency loss mentioned above. As in the typical semi-\nsupervised learning methods [13, 14], ramp-up and ramp-down techniques, which can be de\ufb01ned by\nthe weight scheduling w(t), are used for the stable training.\n\nL = Lc + Ll + w(t) \u00b7 Lcon\n\n3.4 Application to two-stage detector\n\nUnlike the single-stage detector, the two-stage detector has region proposal network (RPN) to generate\nregion proposals and recognize the objectness of them. If we pass both the original and the \ufb02ipped\nimages to the RPN, the correspondence matching problem between the region proposals occurs\nwhich is relatively hard to solve. To simplify the problem, we only pass the feature \u03c6(I) generated\nfrom the original image to the RPN. Then the output RoI locations are reversed and applied to the\ncorresponding feature \u03c6( \u02c6I) as shown in Fig. 2 (b). Given the feature map \u03c6( \u02c6I) from the backbone\nnetwork and the k-th RoI, hk, from the RPN, the RoI-speci\ufb01c feature map of \u02c6I corresponding\nto the \ufb02ipped area \u02c6hk can be easily derived. As shown in Fig. 2 (b) RPN is trained without the\nconsistency loss. The features corresponding to the RoI, r(\u03c6(I), hk) and r(\u03c6( \u02c6I), \u02c6hk), are processed\nby a classi\ufb01er g. Then, outputs g(r(\u03c6(I), hk)) and g(r(\u03c6( \u02c6I), \u02c6hk)) are used to compute the loss to\ntrain the network. As will be seen in the experiments, compared to the single-stage detector, the\nperformance improvement of the proposed CSD is lower for two-stage detector and this attributes to\nthe lack of consistency loss in RPN training.\n\n3.5 Background elimination\n\nParticularly in object detection, an additional class of \u2018background\u2019 exists and most of the candidate\nboxes are usually classi\ufb01ed to this class unless it is \ufb01ltered by a con\ufb01dence threshold. Consequently,\nconsistency losses computed with all candidates will be easily dominated by backgrounds. This can\ndegrade the classi\ufb01cation performance for the foreground classes. Therefore, we exclude boxes having\na high probability of background class by marking it with a mask. The mask is created according to\nthe classi\ufb01cation result for every candidate bounding box of I as in\n\n(cid:26)1,\n\nif argmax(f k\n\ncls(I)) (cid:54)= background\n\nmk =\n\n0, otherwise.\n\nApplying the mask to (2) and (4) yields\ncls(I), f k(cid:48)\n\nLcon\u2212c = EI\n\n[lcon_loc(f k\nwhere Imk=1 indicates that the expectation is taken only for the positive mask.\n\n[lcon_cls(f k\n\nmk =1\n\nmk =1\n\ncls( \u02c6I))], Lcon\u2212l = EI\n\nloc(I), f k(cid:48)\n\nloc( \u02c6I))]\n\n5\n\n\fTable 1: Detection results for PASCAL VOC2007 test set. The \ufb01rst two rows show the performance\nof each detector by supervised learning. * is the score from [17, 18]. The following experiments use\nVOC07 as the labeled data and VOC12 as the unlabeled data, and show the results of the proposed\nCSD with/without Lcon\u2212c (cls), Lcon\u2212l (loc) and EB. Blue / Red : supervised score (baseline) and\nBest results. The numbers in the parentheses are the performance enhancement over the baseline.\n\nLabeled\n\ndata\n\nVOC07\nVOC0712\n\nUnlabeled\n\ndata\n\n-\n-\n\nVOC07\n\nVOC12\n\nVOC07\n\nVOC12\n\n4 Experiments\n\nConsistency Background\ncls\nElimination\n-\n-\n(cid:88)\n-\n(cid:88)\n(cid:88)\n-\n(cid:88)\n\nloc\n-\n-\n-\n(cid:88)\n(cid:88)\n-\n(cid:88)\n(cid:88)\n\n-\n-\n-\n-\n-\n(cid:88)\n(cid:88)\n(cid:88)\n\nSSD 300\n68.0*/70.2\n74.3*/77.2\n71.6 (1.4)\n72.2 (2.0)\n72.0 (1.8)\n71.7 (1.5)\n71.9 (1.7)\n72.3 (2.1)\n\nmAP (%)\nSSD 512\n71.6*/73.3\n76.8*/79.6\n74.6 (1.3)\n74.6 (1.3)\n74.8 (1.5)\n75.4 (2.1)\n75.2 (1.9)\n75.8 (2.5)\n\nR-FCN\n73.9\n\n79.5*/79.4\n74.0 (0.1)\n73.9 (0.0)\n74.0 (0.1)\n74.5 (0.6)\n74.4 (0.5)\n74.7 (0.8)\n\nIn our experiments, we have utilized the PASCAL VOC [22] and MSCOCO [23] datasets which are\nthe most popular datasets in object detection. They consist of 20 and 80 classes respectively. PASCAL\nVOC 2007 and 2012 datasets consist of 5,011 and 11,540 trainval (train and validation) images\nrespectively. In this paper, PASCAL VOC2007 trainval is used as the labeled data and PASCAL\nVOC2012 trainval and MSCOCO are utilized as the unlabeled one. We use test set of PASCAL\nVOC2007 (4,952 images) for testing.\nThe codes used for our experiments are based on Pytorch. We have used third-party codes for SSD\n[17] 4 and R-FCN [18] 5. All experiments have been done under the similar setting with the code6 of\nthe author. Expediently, labeled and unlabeled data are gathered in a single dataset and then randomly\nshuf\ufb02ed. In our setting, both labeled and unlabeled samples sit together in each mini-batch. The\nexperimental settings of R-FCN are referred to those of SSM. As the batch size used for R-FCN is 4,\nusing the same sampling strategy of SSD experiments does not guarantee that at least one labeled\ndata is included in every mini-batch. To solve this problem, we have established separate data-loaders\nfor labeled and unlabeled data. The amount of unlabeled data in a mini-batch is three times larger\nthan that of the labeled data7. The total number of RoIs for CSD is 2k and all the parameter settings\nand training details are presented in the supplementary material.\n\n4.1 Ablation Study\nWe have examined the in\ufb02uence of Lcon\u2212c, Lcon\u2212l and Background Elimination (BE) on SSD300,\nSSD512 and R-FCN and the performances are presented in Table 1. For SSD300, supervised learning\nusing VOC07 and VOC0712 show 70.2 mAP and 77.2 mAP respectively. Using Lcon\u2212c with Jensen-\nShannon divergence induces 1.4% of improvement while Lcon\u2212c with L2-norm causes a performance\ndegradation to 70.0 mAP, which is slightly lower than that of the supervised learning. Lcon\u2212l\nshows 2.0% of enhancement and jointly using both consistency losses shows 1.8% of enhancement.\nParticularly in SSD300 using Lcon\u2212l only has shown better performance than using both. SSD512\nscored 73.3 mAP and 79.6 mAP in pure supervised learning on VOC07 and VOC0712 respectively.\nSeparate use of Lcon\u2212c or Lcon\u2212l induces 1.3% of improvements in both cases and joint usage\nof both losses improves 1.5% of accuracy. BE signi\ufb01cantly improves the performance when used\nwith both of the consistency losses. Especially, since more regions are predicted as backgrounds in\nSSD512 compared to SSD300, BE is more bene\ufb01cial to SSD512 than to SSD300.\nAs mentioned in section 3.4, CSD in R-FCN uses consistency losses only after the RoI pooling and\nnot in the RPN. For R-FCN, supervised learning using VOC07 and VOC0712 shows 73.9 mAP\nand 79.4 mAP of accuracy respectively. There are small or no performance improvement before\n\n4https://github.com/amdegroot/ssd.pytorch\n5https://github.com/princewang1994/R-FCN.pytorch\n6https://github.com/weiliu89/caffe/tree/ssd\n7During the training, we allow the labeled data and unlabeled data not to share the same epoch number.\n\n6\n\n\fTable 2: Detection results on PASCAL VOC2007 test set.\n\u201cCOCO\u00a7\u201d: All 80 classes. \u201cCOCO\u2020\u201d: 20 PASCAL VOC classes.\n\nLabeled\n\ndata\n\nVOC07\n\nVOC07\n\nUnlabeled\n\ndata\n\n-\n\nVOC12\n\nVOC12+COCO\u00a7\nVOC12+COCO\u2020\n\nCSD Method (mAP)\n\nSSD512 R-FCN\n73.9\n74.7\n74.9\n75.1\n\n73.3\n75.8\n75.1\n75.9\n\nSSD300\n\n70.2\n72.3\n71.7\n72.6\n\nTable 3: Effects of using Back-\nground Elimination (BE) on\nSSD300 performance.\n\nVOC07(L)+VOC12(U) mAP\nwithout BE\n72.0\n72.3\nBE with mk\nBE with mk \u2297 mk(cid:48)\n71.7\n\napplying BE. However, when BE is applied, performance is improved by adding Lcon\u2212c and Lcon\u2212l.\nIn addition, the performance is further improved with simultaneous use of both consistency losses.\n\n4.2 Unlabeled data with different distribution (MSCOCO)\n\nTo see the effect of unlabeled data with different distribution to the labeled set, we use VOC07 as\nthe labeled data and VOC12 plus MSCOCO as the unlabeled data as shown in Table 2. We denote\n\u2018trainval\u2019 of the MSCOCO dataset (123,287 images) as COCO\u00a7 and the dataset (19,592 images) of\nwhich images contain only objects belonging to the 20 PASCAL VOC classes as COCO\u2020. Details on\nlearning scheduling are in the supplementary material.\nIn single-stage detectors, the performance by training with unlabeled VOC12 and COCO\u00a7 shows\nbetter performance than the supervised learning, but it is less than the performance using unlabeled\nVOC12 only. In a two stage detector, it shows higher performance than the supervised learning and\ntraining with unlabeled VOC12 data. Training with unlabeled VOC12 and COCO\u2020, both the single-\nstage detector and two-stage detector show performance improvements. We analyze this phenomenon\nin the next section.\n\n5 Discussion\n\nConsistency regularization with only labeled data: We evaluated our method on PASCAL VOC\n2007 under the supervised training setting. We observed that training with the consistency loss only\non labeled data led to worse results. It means that the consistency loss does not affect the improvement\nof performance for labeled data. Our consistency loss has a regularization effect on the CNN \ufb01lters\nsuch that they are enforced to be symmetric. We conjecture this reduces the representation power of\nCNNs and causes performance degradation when used with ground truth labels. However, as shown\nin Table 1, our consistency constraints are helpful to improve the performance for semi-supervised\nobject detection task. The results of these experiments are provided in the Supplementary Material.\nSingle-stage detector vs. Two-stage detector: We apply consistency constraint differently depend-\ning on whether RPN is used or not. First, in a single-stage detector, the proposed consistency losses\ncan be applied to all areas and it shows much improvement in performance. The two-stage detector,\non the other hand, uses \u02c6h by \ufb02ipping the h obtained from I. Therefore, while we can expect to\nimprove performance in the classi\ufb01er, it is hard to expect additional performance improvement of\nRPN. As a result, the two-stage detector has less performance improvement than the single-stage\ndetector. To optimize the RPN, a new way exploiting the consistency loss is needed, which we leave\nas further work.\nBackground Elimination: The proportion of background in the prede\ufb01ned boxes is very large. We\napply BE to reduce the effect of the background and show that BE is helpful in improving the\nperformance. However, getting rid of too many samples is not helpful in learning, as shown in Table\n3. As a way of reducing more background samples, the consistency losses are applied to the candidate\nboxes only when their estimated class is non-background (mk = 1) as well as their \ufb02ipped boxes on\nthe \ufb02ipped images are estimated as non-background (mk(cid:48)\n= 1). At this time, the performance of the\nSSD300 model shows 71.7 mAP, which is 0.6% lower than the original 72.3 mAP. This shows that\nremoving too many background samples may cause performance degradation.\nDatasets: Table 2 shows that in learning 20 classes of VOC, additional usage of unlabeled data leads\nto an enhanced performance. However, the ratio of labeled/unlabeled class mismatch decides the\n\n7\n\n\fTable 4: Comparisons between self-training and consistency regularization based methods on PAS-\nCAL VOC2007 test set. \u201cCOCO\u00a7\u201d: All 80 classes. \u201cCOCO\u2020\u201d: 20 PASCAL VOC classes.\n\nSingle-Stage Detector\nMethod\nSSD512 (supervised)\nSSD512 + CSD (ours)\nSSD512 + CSD (ours)\nSSD512 + CSD (ours)\nTwo-Stage Detector\nMethod\nR-FCN (supervised)\nRFCN + SPL (300%) [24]\nRFCN + SPL (400%) [24]\nRFCN + SSM (300%) [10]\nRFCN + SSM (400%) [10]\nRFCN + CSD (ours)\nRFCN + CSD (ours)\nRFCN + CSD (ours)\n\nLabeled data\n\nVOC07\nVOC07\nVOC07\nVOC07\n\nLabeled data\n\nVOC07\nVOC07\nVOC07\nVOC07\nVOC07\nVOC07\nVOC07\nVOC07\n\nUnlabeled data\n\n-\n\nVOC12\n\nVOC12 + COCO\u00a7\nVOC12 + COCO\u2020\n\nUnlabeled data\n\n-\n\nVOC12 + COCO\u00a7\n\nVOC12\n\nVOC12 + COCO\u00a7\nVOC12 + COCO\u2020\n\nmAP Gain\n73.3\n75.8\n75.1\n75.9\n\n-\n2.5\n1.8\n2.6\n\nmAP Gain\n73.9\n74.1\n74.7\n75.6\n76.7\n74.7\n74.9\n75.1\n\n-\n0.2\n0.8\n1.7\n2.8\n0.8\n1.0\n1.2\n\namount of improvement. This is why the case of using VOC12 + COCO\u2020 shows a better result than\nthe case of VOC12 + COCO\u00a7. This result is consistent with the recent study by [16].\nBE is hardly expected to remove this out-of-distribution. It is intended to eliminate background, but\nclasses in MSCOCO can have a higher con\ufb01dence in other classes that are similar. For example,\nclasses such as \u2018giraffe\u2019 and \u2018elephant\u2019 may have features similar to \u2018horse\u2019 or \u2018dog\u2019 rather than the\nbackground. These data can interfere with training detectors.\nOn the other hand, adding unlabeled data with a similar distribution, all detectors have improved the\nperformance. Our CSD does not need any labeling in the additional data but it still has its limitation\nthat the distribution of the unlabeled data should be similar to that of the labeled data. Further research\nis needed to solve this problem, which we leave it for future work.\nSelf-training vs. Consistency regularization: Self-training is widely used as a simple heuristic\nmethod in semi-supervised learning. As it is an iterative method which cycles training, prediction of\nunlabeled data and changing the training dataset, it is time-consuming and computationally intensive\n[12]. In addition, the threshold and stop criterion, which decide the quality and quantity of an\nadditional dataset, affect the algorithm\u2019s performance. Meanwhile, CR method which trains unlabeled\ndata with an additional loss helps the more common and robust learning.\nTable 4 shows the performance of SPL [24], SSM [10] and CSD. SPL and SSM are based on the\nself-training method, which shows different performance depending on the amount of data added 8.\nThey experimented only with R-FCN framework and trained with VOC07 as labeled data and VOC12\nand MSCOCO as unlabeled data. The performance of SPL is improved by 0.2 and 0.8 than baseline\nwhile SSM has 1.7 and 2.8 better performance than the baseline. In CSD, according to unlabeled\ndataset, it shows performance improvement of 0.8 \u223c 1.2 than baseline. As mentioned above, CSD\nhas a limitation in the two stage detector, which has less performance improvement than single\nstage detector. In single stage detector, however, SSD512 shows the 1.8% and 2.6% performance\nimprovements.\n\n6 Conclusion\n\nIn this paper, we have introduced a novel Consistency-based Semi-supervised learning for object\nDetection (CSD) method. To the best of our knowledge, it is the \ufb01rst attempt to extend CR used in\nconventional semi-supervised classi\ufb01cation problems to object detection problem. We applied the\nproposed CSD to single-stage detectors and a two-stage detector respectively and designed loss to\nimprove the performance of both detectors over the supervised learning method. We have shown that\nconsistency loss is helpful for semi-supervised learning in classi\ufb01cation as well as localization with\nvarious ablation experiments. In addition, BE has been shown to improve performance.\n\n8% means that the percentage of additional unlabeled objects over labeled objects.\n\n8\n\n\f7 Acknowledgments\n\nThis work was supported by IITP grant funded by the Korea government (MSIT) (No.2019-0-01367)\nand Next-Generation Information Computing Development Program through the NRF of Korea\n(2017M3C4A7077582).\n\nReferences\n[1] Olga Russakovsky, Li-Jia Li, and Li Fei-Fei. Best of both worlds: human-machine collaboration\nfor object annotation. In Proceedings of the IEEE conference on computer vision and pattern\nrecognition, pages 2121\u20132131, 2015.\n\n[2] Amy Bearman, Olga Russakovsky, Vittorio Ferrari, and Li Fei-Fei. What\u2019s the point: Semantic\nIn European conference on computer vision, pages\n\nsegmentation with point supervision.\n549\u2013565. Springer, 2016.\n\n[3] Piotr Dollar, Christian Wojek, Bernt Schiele, and Pietro Perona. Pedestrian detection: An\nevaluation of the state of the art. IEEE transactions on pattern analysis and machine intelligence,\n34(4):743\u2013761, 2012.\n\n[4] Yi Zhu, Yanzhao Zhou, Qixiang Ye, Qiang Qiu, and Jianbin Jiao. Soft proposal networks for\nweakly supervised object localization. In Proceedings of the IEEE International Conference on\nComputer Vision, pages 1841\u20131850, 2017.\n\n[5] Miaojing Shi, Holger Caesar, and Vittorio Ferrari. Weakly supervised object localization using\nthings and stuff transfer. In Proceedings of the IEEE International Conference on Computer\nVision, pages 3381\u20133390, 2017.\n\n[6] Zequn Jie, Yunchao Wei, Xiaojie Jin, Jiashi Feng, and Wei Liu. Deep self-taught learning for\nweakly supervised object localization. In Proceedings of the IEEE Conference on Computer\nVision and Pattern Recognition, pages 1377\u20131385, 2017.\n\n[7] Jiajie Wang, Jiangchao Yao, Ya Zhang, and Rui Zhang. Collaborative learning for weakly\n\nsupervised object detection. arXiv preprint arXiv:1802.03531, 2018.\n\n[8] Yuxing Tang, Josiah Wang, Boyang Gao, Emmanuel Dellandr\u00e9a, Robert Gaizauskas, and Liming\nChen. Large scale semi-supervised object detection using visual and semantic knowledge\ntransfer. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,\npages 2119\u20132128, 2016.\n\n[9] Ziang Yan, Jian Liang, Weishen Pan, Jin Li, and Changshui Zhang. Weakly-and semi-supervised\nobject detection with expectation-maximization algorithm. arXiv preprint arXiv:1702.08740,\n2017.\n\n[10] Keze Wang, Xiaopeng Yan, Dongyu Zhang, Lei Zhang, and Liang Lin. Towards human-machine\ncooperation: Self-supervised sample mining for object detection. In Proceedings of the IEEE\nConference on Computer Vision and Pattern Recognition, pages 1605\u20131613, 2018.\n\n[11] Nhu-Van Nguyen, Christophe Rigaud, and Jean-Christophe Burie. Semi-supervised object\ndetection with unlabeled data. In international conference on computer vision theory and\napplications, 2019.\n\n[12] Chuck Rosenberg, Martial Hebert, and Henry Schneiderman. Semi-supervised self-training of\n\nobject detection models. 2005.\n\n[13] Samuli Laine and Timo Aila. Temporal ensembling for semi-supervised learning. arXiv preprint\n\narXiv:1610.02242, 2016.\n\n[14] Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averaged\nconsistency targets improve semi-supervised deep learning results. In Advances in neural\ninformation processing systems, pages 1195\u20131204, 2017.\n\n9\n\n\f[15] Takeru Miyato, Shin-ichi Maeda, Shin Ishii, and Masanori Koyama. Virtual adversarial training:\na regularization method for supervised and semi-supervised learning. IEEE transactions on\npattern analysis and machine intelligence, 2018.\n\n[16] Avital Oliver, Augustus Odena, Colin A Raffel, Ekin Dogus Cubuk, and Ian Goodfellow.\nIn Advances in Neural\n\nRealistic evaluation of deep semi-supervised learning algorithms.\nInformation Processing Systems, pages 3235\u20133246, 2018.\n\n[17] Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng-Yang\nFu, and Alexander C Berg. Ssd: Single shot multibox detector. In European conference on\ncomputer vision, pages 21\u201337. Springer, 2016.\n\n[18] Jifeng Dai, Yi Li, Kaiming He, and Jian Sun. R-fcn: Object detection via region-based fully\nconvolutional networks. In Advances in neural information processing systems, pages 379\u2013387,\n2016.\n\n[19] Olivier Chapelle, Bernhard Scholkopf, and Alexander Zien. Semi-supervised learning (chapelle,\no. et al., eds.; 2006)[book reviews]. IEEE Transactions on Neural Networks, 20(3):542\u2013542,\n2009.\n\n[20] Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: Uni\ufb01ed,\nreal-time object detection. In Proceedings of the IEEE conference on computer vision and\npattern recognition, pages 779\u2013788, 2016.\n\n[21] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object\ndetection with region proposal networks. In Advances in neural information processing systems,\npages 91\u201399, 2015.\n\n[22] Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman.\nThe pascal visual object classes (voc) challenge. International journal of computer vision,\n88(2):303\u2013338, 2010.\n\n[23] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr\nDoll\u00e1r, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European\nconference on computer vision, pages 740\u2013755. Springer, 2014.\n\n[24] M Pawan Kumar, Benjamin Packer, and Daphne Koller. Self-paced learning for latent variable\n\nmodels. In Advances in Neural Information Processing Systems, pages 1189\u20131197, 2010.\n\n10\n\n\f", "award": [], "sourceid": 5738, "authors": [{"given_name": "Jisoo", "family_name": "Jeong", "institution": "Seoul National University"}, {"given_name": "Seungeui", "family_name": "Lee", "institution": "Seoul National University"}, {"given_name": "Jeesoo", "family_name": "Kim", "institution": "Seoul National University"}, {"given_name": "Nojun", "family_name": "Kwak", "institution": "Seoul National University"}]}