{"title": "Do Deep Neural Networks Suffer from Crowding?", "book": "Advances in Neural Information Processing Systems", "page_first": 5628, "page_last": 5638, "abstract": "Crowding is a visual effect suffered by humans, in which an object that can be recognized in isolation can no longer be recognized when other objects, called flankers, are placed close to it. In this work, we study the effect of crowding in artificial Deep Neural Networks (DNNs) for object recognition. We analyze both deep convolutional neural networks  (DCNNs) as well as an extension of DCNNs that are multi-scale and that change the receptive field size of the convolution filters with their position in the image.  The latter  networks, that we call eccentricity-dependent, have been proposed for modeling the feedforward path of the primate visual cortex. Our results reveal that the eccentricity-dependent model, trained on target objects in isolation, can recognize such targets in the presence of flankers, if the targets are near the center of the image, whereas DCNNs cannot. Also, for all tested networks, when trained on targets in isolation, we find that recognition accuracy of the networks decreases the closer the flankers are to the target and the more flankers there are. We find that visual similarity between the target and flankers also plays a role and that pooling in early layers of the network leads to more crowding. Additionally, we show that incorporating flankers  into the images of the training set for learning the DNNs does not lead to robustness against configurations not seen at training.", "full_text": "Do Deep Neural Networks Suffer from Crowding?\n\nAnna Volokitin\u2020(cid:92)\n\nGemma Roig\u2020\u2021\u03b9\ngemmar@mit.edu\n\nTomaso Poggio\u2020\u2021\ntp@csail.mit.edu\n\nvoanna@vision.ee.ethz.ch\n\u2020Center for Brains, Minds and Machines, Massachusetts Institute of Technology, Cambridge, MA\n\n\u2021Istituto Italiano di Tecnologia at Massachusetts Institute of Technology, Cambridge, MA\n\n(cid:92)Computer Vision Laboratory, ETH Zurich, Switzerland\n\n\u03b9Singapore University of Technology and Design, Singapore\n\nAbstract\n\nCrowding is a visual effect suffered by humans, in which an object that can be\nrecognized in isolation can no longer be recognized when other objects, called\n\ufb02ankers, are placed close to it. In this work, we study the effect of crowding in\narti\ufb01cial Deep Neural Networks (DNNs) for object recognition. We analyze both\ndeep convolutional neural networks (DCNNs) as well as an extension of DCNNs\nthat are multi-scale and that change the receptive \ufb01eld size of the convolution \ufb01lters\nwith their position in the image. The latter networks, that we call eccentricity-\ndependent, have been proposed for modeling the feedforward path of the primate\nvisual cortex. Our results reveal that the eccentricity-dependent model, trained on\ntarget objects in isolation, can recognize such targets in the presence of \ufb02ankers,\nif the targets are near the center of the image, whereas DCNNs cannot. Also, for\nall tested networks, when trained on targets in isolation, we \ufb01nd that recognition\naccuracy of the networks decreases the closer the \ufb02ankers are to the target and\nthe more \ufb02ankers there are. We \ufb01nd that visual similarity between the target and\n\ufb02ankers also plays a role and that pooling in early layers of the network leads\nto more crowding. Additionally, we show that incorporating \ufb02ankers into the\nimages of the training set for learning the DNNs does not lead to robustness against\ncon\ufb01gurations not seen at training.\n\n1\n\nIntroduction\n\nDespite stunning successes in many computer vision problems [1, 2, 3, 4, 5], Deep Neural Networks\n(DNNs) lack interpretability in terms of how the networks make predictions, as well as how an\narbitrary transformation of the input, such as addition of clutter in images in an object recognition\ntask, will affect the function value.\nExamples of an empirical approach to this problem are testing the network with adversarial exam-\nples [6, 7] or images with different geometrical transformations such as scale, position and rotation,\nas well as occlusion [8]. In this paper, we add clutter to images to analyze the crowding in DNNs.\nCrowding is a well known effect in human vision [9, 10], in which objects (targets) that can be\nrecognized in isolation can no longer be recognized in the presence of nearby objects (\ufb02ankers), even\nthough there is no occlusion. We believe that crowding is a special case of the problem of clutter in\nobject recognition. In crowding studies, human subjects are asked to \ufb01xate at a cross at the center of\na screen, and objects are presented at the periphery of their visual \ufb01eld in a \ufb02ash such that the subject\nhas no time to move their eyes. Experimental data suggests that crowding depends on the distance of\nthe target and the \ufb02ankers [11], eccentricity (the distance of the target to the \ufb01xation point), as well as\nthe similarity between the target and the \ufb02ankers [12, 13] or the con\ufb01guration of the \ufb02ankers around\nthe target object [11, 14, 15].\n\n31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA.\n\n\fwhole image\n\n(a)\n\nMNIST\n\n(b)\n\nnotMNIST\n\n(c)\n\nOmniglot\n\n(d)\n\nPlaces background\n\n(e)\n\nFigure 1: (a) Example image used to test the models, with even MNIST as target and two odd MNIST \ufb02ankers.\n(b-d) Close-up views with odd MNIST, notMNIST and Omniglot datasets as \ufb02ankers, respectively. (e) An even\nMNIST target embedded into a natural image.\n\nMany computational models of crowding have been proposed e.g. [16, 17]. Our aim is not to model\nhuman crowding. Instead, we characterize the crowding effect in DNNs trained for object recognition,\nand analyze which models and settings suffer less from such effects.\nWe investigate two types of DNNs for crowding: traditional deep convolutional neural networks and\nan extension of these which is multi-scale, and called eccentricity-dependent model [18]. Inspired\nby the retina, the receptive \ufb01eld size of the convolutional \ufb01lters in this model grows with increasing\ndistance from the center of the image, called the eccentricity. Cheung et al. [19] explored the\nemergence of such property when the visual system has an eye-\ufb01xation mechanism.\nWe investigate under which conditions crowding occurs in DNNs that have been trained with images\nof target objects in isolation. We test the DNNs with images that contain the target object as well\nas clutter, which the network has never seen at training. Examples of the generated images using\nMNIST [20], notMNIST [21], and Omniglot [22] datasets are depicted in Fig 1, in which even\nMNIST digits are the target objects. As done in human psychophysics studies, we take recognition\naccuracy to be the measure of crowding. If a DNN can recognize a target object correctly despite the\npresence of clutter, crowding has not occurred.\nOur experiments reveal the dependence of crowding on image factors, such as \ufb02anker con\ufb01guration,\ntarget-\ufb02anker similarity, and target eccentricity. Our results also show that prematurely pooling\nsignals increases crowding. This result is related to the theories of crowding in humans. In addition,\nwe show that training the models with cluttered images does not make models robust to clutter and\n\ufb02ankers con\ufb01gurations not seen in training. Thus, training a model to be robust to general clutter is\nprohibitively expensive.\nWe also discover that the eccentricity-dependent model, trained on isolated targets, can recognize\nobjects even in very complex clutter, i.e. when they are embedded into images of places (Fig 1(e)).\nThus, if such models are coupled with a mechanism for selecting eye \ufb01xation locations, they can be\ntrained with objects in isolation being robust to clutter, reducing the amount of training data needed.\n\n2 Models\n\nIn this section we describe the DNN architectures for which we characterize crowding effect. We\nconsider two kinds of DNN models: Deep Convolutional Neural Networks and eccentricity-dependent\nnetworks, each with different pooling strategies across space and scale. We investigate pooling in\nparticular, because we [18, 23] as well as others [24] have suggested that feature integration by\npooling may be the cause of crowding in human perception.\n\n2.1 Deep Convolutional Neural Networks\n\nThe \ufb01rst set of models we investigate are deep convolutional neural networks (DCNN) [25], in which\nthe image is processed by three rounds of convolution and max pooling across space, and then passed\nto one fully connected layer for the classi\ufb01cation. We investigate crowding under three different\nspatial pooling con\ufb01gurations, listed below and shown in Fig 2. The word pooling in the names of\nthe model architectures below refers to how quickly we decrease the feature map size in the model.\nAll architectures have 3\u00d73 max pooling with various strides, and are:\n\n2\n\n\fno total pooling\n\nprogressive pooling\n\nat end pooling\n\nFigure 2: DCNN architectures with three convolutional layers and one fully connected, trained to recognize\neven MNIST digits. These are used to investigate the role of pooling in crowding. The grey arrow indicates\ndownsampling.\n\nFigure 3: Eccentricity-dependent model: Inverted pyramid with sampling points. Each circle\nrepresents a \ufb01lter with its respective receptive \ufb01eld. For simplicity, the model is shown with 3 scales.\n\npooling has stride 1. The square feature maps sizes after each pool layer are 60-54-48-42.\n\n\u2022 No total pooling Feature maps sizes decrease only due to boundary effects, as the 3\u00d73 max\n\u2022 Progressive pooling 3\u00d73 pooling with a stride of 2 halves the square size of the feature maps,\nuntil we pool over what remains in the \ufb01nal layer, getting rid of any spatial information before the\nfully connected layer. (60-27-11-1).\n\u2022 At end pooling Same as no total pooling, but before the fully connected layer, max-pool over the\n\nentire feature map. (60-54-48-1).\n\nThe data in each layer in our model is a 5-dimensional tensor of minibatch size\u00d7 x \u00d7 y \u00d7 number\nof channels, in which x de\ufb01nes the width and y the height of the input. The input image to the\nmodel is resized to 60 \u00d7 60 pixels. In our training, we used minibatches of 128 images, 32 feature\nchannels for all convolutional layers, and convolutional \ufb01lters of size 5 \u00d7 5 and stride 1.\n\n2.2 Eccentricity-dependent Model\n\nThe second type of DNN model we consider is an eccentricity-dependent deep neural network,\nproposed by Poggio et al. in [18] as a model of the human visual cortex and further studied in [23].\nIts eccentricity dependence is based on the human retina, which has receptive \ufb01elds which increase\nin size with eccentricity. [18] argues that the computational reason for this property is the need to\ncompute a scale- and translation-invariant representation of objects. [18] conjectures that this model\nis robust to clutter when the target is near the \ufb01xation point.\nAs discussed in [18], the set of all scales and translations for which invariant representations can be\ncomputed lie within an inverted truncated pyramid shape, as shown in Fig 3(a). The width of the\npyramid at a particular scale is roughly related to the amount of translation invariance for objects\nof that size. Scale invariance is prioritized over translation invariance in this model, in contrast to\n\n3\n\n02468image sampled atdifferent scalesbefore downsamplinginput to modelscaleinverted pyramid samplingfilteryx(a)(b)(c)(d)\fclassical DCNNs. From a biological point of view, the limitation of translation invariance can be\ncompensated for by eye movements, whereas to compensate for a lack of scale invariance the human\nwould have to move their entire body to change their distance to the object.\nThe eccentricity-dependent model computes an invariant representation by sampling the inverted\npyramid at a discrete set of scales with the same number of \ufb01lters at each scale. At larger scales,\nthe receptive \ufb01elds of the \ufb01lters are also larger to cover a larger image area, see Fig 3(a). Thus, the\nmodel constructs a multi-scale representation of the input, where smaller sections (crops) of the\nimage are sampled densely at a high resolution, and larger sections (crops) are sampled with at a\nlower resolution, with each scale represented using the same number of pixels, as shown in Fig 3(b-d).\nEach scale is treated as an input channel to the network and then processed by convolutional \ufb01lters,\nthe weights of which are shared also across scales as well as space. Because of the downsampling of\nthe input image, this is equivalent to having receptive \ufb01elds of varying sizes. These shared parameters\nalso allow the model to learn a scale invariant representation of the image.\nEach processing step in this model consists of convolution-pooling, as above, as well as max pooling\nacross different scales. Scale pooling reduces the number of scales by taking the maximum value\nof corresponding locations in the feature maps across multiple scales. We set the spatial pooling\nconstant using At end pooling, as described above. The type of scale pooling is indicated by writing\nthe number of scales remaining in each layer, e.g. 11-1-1-1-1. The three con\ufb01gurations tested for\nscale pooling are (1) at the beginning, in which all the different scales are pooled together after the\n\ufb01rst layer, 11-1-1-1-1 (2) progressively, 11-7-5-3-1 and (3) at the end, 11-11-11-11-1, in which all\n11 scales are pooled together at the last layer.\nThe parameters of this model are the same as for the DCNN explained above, except that now there\nare extra \ufb01lters for the scales. Note that because of weight sharing across scales, the number of\nparameters in the eccentricity dependent model is equal that in a standard DCNN. We use 11 crops,\nwith the smallest crop of 60 \u00d7 60 pixels, increasing by a factor of\n2. Exponentially interpolated\ncrops produce fewer boundary effects than linearly interpolated crops, while having qualitatively\nthe same behavior. Results with linearly extracted crops are shown in Fig 7 of the supplementary\nmaterial. All the crops are resized to 60 \u00d7 60 pixels, which is the same input image size used for the\nDCNN above. Image crops are shown in Fig 9.\nContrast Normalization We also investigate the effect of input normalization so that the sum of\nthe pixel intensities in each scale is in the same range. To de-emphasize the smaller crops, which\nwill have the most non-black pixels and therefore dominate the max-pooling across scales, in some\n\u221a\nexperiments we rescale all the pixel intensities to the [0, 1] interval, and then divide them by factor\nproportional to the crop area ((\n\n2)11\u2212i, where i = 1 for the smallest crop).\n\n\u221a\n\n3 Experimental Set-up\n\nModels are trained with back-propagation to recognize a set of objects, which we call targets. During\ntesting, we present the models with images which contain a target object as well as other objects\nwhich the model has not been trained to recognize, which we call \ufb02ankers. The \ufb02anker acts as clutter\nwith respect to the target object.\nSpeci\ufb01cally, we train our models to recognize even MNIST digits\u2014i.e. numbers 0, 2, 4, 6, 8\u2014shifted\nat different locations of the image along the horizontal axis, which are the target objects in our\nexperiments. We compare performance when we use images with the target object in isolation, or\nwhen \ufb02ankers are also embedded in the training images. The \ufb02ankers are selected from odd MNIST\ndigits, notMNIST dataset [21] which contains letters of different typefaces, and Omniglot [22] which\nwas introduced for one-shot character recognition. Also, we evaluate recognition when the target is\nembedded to images of the Places dataset [26].\nThe images are of size 1920 squared pixels, in which we embedded target objects of 120 squared px,\nand \ufb02ankers of the same size, unless contrary stated. Recall that the images are resized to 60 \u00d7 60\nas input to the networks. We keep the training and testing splits provided by the MNIST dataset,\nand use it respectively for training and testing. We illustrate some examples of target and \ufb02anker\ncon\ufb01guration in Fig 1. We refer to the target as a and to the \ufb02anker as x and use this shorthand in the\nplots. All experiments are done in the right half of the image plane. We do this to check if there is a\ndifference between central and peripheral \ufb02ankers. We test the models under 4 conditions:\n\n4\n\n\f\u2022 No \ufb02ankers. Only the target object. (a in the plots)\n\u2022 One central \ufb02anker closer to the center of the image than the target. (xa)\n\u2022 One peripheral \ufb02anker closer to the boundary of the image that the target. (ax)\n\u2022 Two \ufb02ankers spaced equally around the target, being both the same object, see Fig 1. (xax).\n\n4 Experiments\n\nIn this section, we investigate the crowding effect in DNNs. We \ufb01rst carry out experiments on models\nthat have been trained with images containing both targets and \ufb02ankers. We then repeat our analysis\nwith the models trained with images of the targets in isolation, shifted at all positions in the horizontal\naxis. We analyze the effect of \ufb02anker con\ufb01guration, \ufb02anker dataset, pooling in the model architecture,\nand model type, by evaluating accuracy recognition of the target objects.1\n\n4.1 DNNs Trained with Target and Flankers\n\nIn this setup we trained DNNs with images in which there were two identical \ufb02ankers randomly\nchosen from the training set of MNIST odd digits, placed at a distance of 120 pixels on either side\nof the target (xax). The target is shifted horizontally, while keeping the distance between target and\n\ufb02ankers constant, called the constant spacing setup, and depicted in Fig 1(a) of the supplementary\nmaterial. We evaluate (i) DCNN with at the end pooling, and (ii) eccentricity-dependent model with\n11-11-11-11-1 scale pooling, at the end spatial pooling and contrast normalization. We report the\nresults using the different \ufb02anker types at test with xax, ax, xa and a target \ufb02anker con\ufb01guration, in\nwhich a represents the target and x the \ufb02anker, as described in Section 3. 2\nResults are in Fig 4. In the plots with 120 px spacing, we see that the models are better at recognizing\nobjects in clutter than isolated objects for all image locations tested, especially when the con\ufb01guration\nof target and \ufb02anker is the same at the training images than in the testing images (xax). However, in\nthe plots where target-\ufb02anker spacing is 240 px recognition accuracy falls to less than the accuracy of\nrecognizing isolated target objects. Thus, in order for a model to be robust to all kinds of clutter, it\nneeds to be trained with all possible target-\ufb02anker con\ufb01gurations, which is infeasible in practice.\nInterestingly, we see that the eccentricity model is much better at recognizing objects in isolation\nthan the DCNN. This is because the multi-scale crops divide the image into discrete regions, letting\nthe model learn from image parts as well as the whole image.\nWe performed an additional experiment training the network with images that contain the same\ntarget-\ufb02anker con\ufb01guration as above (xax), but with different spacings between the target and the\n\ufb02ankers, including different spacings on either side of the target. Left spacing and right spacing are\nsampled from 120 px, 240 px and 480 px independently. Train and test images shown in Fig. 2 of\nsupplementary material.\nWe test two conditions: (1) With \ufb02ankers on both sides from the target (xax) at a spacing not seen\nin the training set (360 px); (2) With 360 px spacing, including 2 \ufb02ankers on both sides (4 \ufb02ankers\ntotal, xxaxx). In Fig 5, we show the accuracy for images with 360 px target-\ufb02anker spacing, and\nsee that accuracy is not impaired, neither for the DCNN nor the eccentricity model. Yet, the DCNN\naccuracy for images with four \ufb02ankers is impaired, while the eccentricity model still has unimpaired\nrecognition accuracy provided that the target is in the center of the image.\nThus, the recognition accuracy is not impaired for all tested models when \ufb02ankers are in a similar\ncon\ufb01guration in testing as in training. This is even when the \ufb02ankers at testing are placed at a spacing\nthat is in between two seen spacings used at training. The models can interpolate to new spacings of\n\ufb02ankers when using similar con\ufb01gurations in test images as seen during training, e.g. (xax), arguably\ndue to the pooling operators. Yet, DCNN recognition is still severely impaired and do not generalize\nto new \ufb02anker con\ufb01gurations, such as adding more \ufb02ankers, when there are 2 \ufb02ankers on both sides of\nthe target (xxaxx). To gain robustness to such con\ufb01gurations, each of these cases should be explicitly\nincluded in the training set. Only the eccentricity-dependent model is robust to different \ufb02anker\n\n1Code to reproduce experiments is available at https://github.com/CBMM/eccentricity\n2The ax \ufb02anker line starts at 120 px of target eccentricity, because nothing was put at negative eccentricities.\nFor the case of 2 \ufb02ankers, when the target was at 0-the image center, the \ufb02ankers were put at -120 px and 120 px.\n\n5\n\n\fTest Flankers:\n\nodd MNIST\n\nnotMNIST\n\nomniglot\n\nDCNN\nconstant spacing\nof 120 px\n\nEcc.-dependent model\nconstant spacing\nof 120 px\n\nDCNN\nconstant spacing\nof 240 px\n\nEcc.-dependent model\nconstant spacing\nof 240 px\n\nFigure 4: Even MNIST accuracy recognition of DCNN (at the end pooling) and Eccentricity Model (11-11-11-\n11-1, At End spatial pooling with contrast normalization) trained with odd MNIST \ufb02ankers at 120px constant\nspacing. The target eccentricity is in pixels.\n\nModel:\n\nDCNN\n\nEcc w. Contrast Norm Ecc no Contrast Norm\n\nMNIST \ufb02ankers\n120, 240, 480 px\ndiff. left right spacing\n\nFigure 5: All models tested at 360 px target-\ufb02anker spacing. All models can recognize digit in the presence\nof clutter at a spacing that is in between spacings seen at training time. However, the eccentricity Model\n(11-11-11-11-1, At End spatial pooling with contrast normalization) and the DCNN fail to generalize to new\ntypes of \ufb02anker con\ufb01gurations (two \ufb02ankers on each side, xxaxx) at 360 px spacing between the target and inner\n\ufb02anker\n\ncon\ufb01gurations not included in training, when the target is centered. We will explore the role of\ncontrast normalization in Sec 4.3.\n\n4.2 DNNs Trained with Images with the Target in Isolation\n\nFor these experiments, we train the models with the target object in isolation and in different positions\nof the image horizontal axis. We test the models on images with target-\ufb02anker con\ufb01gurations a, ax,\nxa, xax.\nDCNN We examine the crowding effect with different spatial pooling in the DCNN hierarchy: (i) no\ntotal pooling, (ii) progressive pooling, (iii) at end pooling (see Section 2.1 and Fig 2).\n\n6\n\n\fSpatial Pooling:\n\nNo Total Pooling\n\nProgressive\n\nAt End\n\nMNIST \ufb02ankers\nConstant spacing\n120 px spacing\n\nMNIST \ufb02ankers\nConstant target ecc.\n0 px target ecc.\n\nFigure 6: Accuracy results of 4 layer DCNN with different pooling schemes trained with targets shifted across\nimage and tested with different \ufb02anker con\ufb01gurations. Eccentricity is in pixels.\n\nFlanker dataset:\n\nMNIST\n\nnotMNIST\n\nOmniglot\n\nDCNN\nConstant spacing\n120 px spacing\n\nFigure 7: Effect in the accuracy recognition in DCNN with at end pooling, when using different \ufb02anker datasets\nat testing.\n\nResults are shown in Fig 6. In addition to the constant spacing experiment (see Section 4.1), we also\nevaluate the models in a setup called constant target eccentricity, in which we have \ufb01xed the target in\nthe center of the visual \ufb01eld, and change the spacing between the target and the \ufb02anker, as shown in\nFig 1(b) of the supplementary material. Since the target is already at the center of the visual \ufb01eld,\na \ufb02anker can not be more central in the image than the target. Thus, we only show x, ax and xax\nconditions.\nFrom Fig 6, we observe that the more \ufb02ankers are present in the test image, the worse recognition\ngets. In the constant spacing plots, we see that recognition accuracy does not change with eccentricity,\nwhich is expected, as translation invariance is built into the structure of convolutional networks. We\nattribute the difference between the ax and xa conditions to boundary effects. Results for notMNIST\nand Omniglot \ufb02ankers are shown in Fig 4 of the supplementary material.\nFrom the constant target eccentricity plots, we see that as the distance between target and \ufb02anker\nincreases, the better recognition gets. This is mainly due to the pooling operation that merges the\nneighboring input signals. Results with the target at the image boundary is shown in Fig 3 of the\nsupplementary material.\nFurthermore, we see that the network called no total pooling performs worse in the no \ufb02anker setup\nthan the other two models. We believe that this is because pooling across spatial locations helps the\nnetwork learn invariance. However, in the below experiments, we will see that there is also a limit to\nhow much pooling across scales of the eccentricity model improves performance.\nWe test the effect of \ufb02ankers from different datasets evaluating DCNN model with at end pooling\nin Fig 7. Omniglot \ufb02ankers crowd slightly less than odd MNIST \ufb02ankers. The more similar the\n\ufb02ankers are to the target object\u2014even MNIST, the more recognition impairment they produce. Since\nOmniglot \ufb02ankers are visually similar to MNIST digits, but not digits, we see that they activate the\nconvolutional \ufb01lters of the model less than MNIST digits, and hence impair recognition less.\n\n7\n\n\fScale Pooling:\n\n11-1-1-1-1\n\n11-7-5-3-1\n\n11-11-11-11-1\n\nNo contrast norm.\nConstant spacing\n120 px spacing\n\nWith contrast norm.\nConstant spacing\n120 px spacing\n\nFigure 8: Accuracy performance of Eccentricity-dependent model with spatial At End pooling, and changing\ncontrast normalization and scale pooling strategies. Flankers are odd MNIST digits.\n\nWe also observe that notMNIST \ufb02ankers crowd much more than either MNIST or Omniglot \ufb02ankers,\neven though notMNIST characters are much more different to MNIST digits than Omniglot \ufb02ankers.\nThis is because notMNIST is sampled from special font characters and these have many more edges\nand white image pixels than handwritten characters. In fact, both MNIST and Omniglot have about\n20% white pixels in the image, while notMNIST has 40%. Fig 5 of the supplementary material\nshows the histogram of the \ufb02anker image intensities. The high number of edges in the notMNIST\ndataset has a higher probability of activating the convolutional \ufb01lters and thus in\ufb02uencing the \ufb01nal\nclassi\ufb01cation decision more, leading to more crowding.\nEccentricity Model We now repeat the above experiment with different con\ufb01gurations of the eccen-\ntricity dependent model. In this experiment, we choose to keep the spacial pooling constant (at end\npooling), and investigate the effect of pooling across scales, as described in Section 2.2. The three\ncon\ufb01gurations for scale pooling are (1) at the beginning, (2) progressively and (3) at the end. The\nnumbers indicate the number of scales at each layer, so 11-11-11-11-1 is a network in which all 11\nscales have been pooled together at the last layer.\nResults with odd MNIST \ufb02ankers are shown in Fig 8. Our conclusions for the effect of the \ufb02anker\ndataset are similar to the experiment above with DCNN. (Results with other \ufb02anker datasets shown in\nFig 6 of the supplementary material.)\nIn this experiment, there is a dependence of accuracy on target eccentricity. The model without\ncontrast normalization is robust to clutter at the fovea, but cannot recognize cluttered objects in the\nperiphery. Interestingly, also in psychophysics experiments little effect of crowding is observed at the\nfovea [10]. The effect of adding one central \ufb02anker (ax) is the same as adding two \ufb02ankers on either\nside (xax). This is because the highest resolution area in this model is in the center, so this part of the\nimage contributes more to the classi\ufb01cation decision. If a \ufb02anker is placed there instead of a target,\nthe model tries to classify the \ufb02anker, and, it being an unfamiliar object, fails. The dependence of\naccuracy on eccentricity can however be mitigated by applying contrast normalization. In this case,\nall scales contribute equally in contrast, and dependence of accuracy on eccentricity is removed.\nFinally, we see that if scale pooling happens too early in the model, such as in the 11-1-1-1-1\narchitecture, there is more crowding. Thus, pooling too early in the architecture prevents useful\ninformation from being propagated to later processing in the network. For the rest of the experiments,\nwe always use the 11-11-11-11-1 con\ufb01guration of the model with spatial pooling at the end.\n\n4.3 Complex Clutter\n\nPrevious experiments show that training with clutter does not give robustness to clutter not seen in\ntraining, e.g. more or less \ufb02ankers, or different spacing. Also, that the eccentricity-dependent model is\nmore robust to clutter when the target is at the image center and no contrast normalization is applied,\nFig 8. To further analyze the models robustness to other kinds of clutter, we test models trained\n\n8\n\n\f(a) crop outlines\n\n(b) crops resampled\n\n(c) results on places images\n\nFigure 9: (a-b) An example of how multiple crops of an input image look, as well as (c) recognition accuracy\nwhen MNIST targets are embedded into images of places.\n\non images with isolated targets shifted along the horizontal axis, with images in which the target is\nembedded into randomly selected images of Places dataset [26], shown in Fig 1(e) and Fig 9(a), (b).\nWe tested the DCNN and the eccentricity model (11-11-11-11-1) with and without contrast normal-\nization, both with at end pooling. The results are in Fig 9(c): only the eccentricity model without\ncontrast normalization can recognize the target and only when the target is close to the image center.\nThis implies that the eccentricity model is robust to clutter: it doesn\u2019t need to be trained with all\ndifferent kinds of clutter. If it can \ufb01xate on the relevant part of the image, it can still discriminate the\nobject, even at multiple object scales because this model is scale invariant [18].\n\n5 Discussion\n\nWe investigated whether DNNs suffer from crowding, and if so, under which conditions, and what\ncan be done to reduce the effect. We found that DNNs suffer from crowding. We also explored the\nmost obvious approach to mitigate this problem, by including clutter in the training set of the model.\nYet, this approach does not help recognition in crowding, unless, of course, a similar con\ufb01guration of\nclutter is used for training and testing.\nWe explored conditions under which DNNs trained with images of targets in isolation are robust to\nclutter. We trained various architectures of both DCNNs and eccentricity-dependent models with\nimages of isolated targets, and tested them with images containing a target at varying image locations\nand 0, 1 or 2 \ufb02ankers, as well as with the target object embedded into complex scenes. We found the\nfour following factors in\ufb02uenced the amount of crowding in the models:\n\u2022 Flanker Con\ufb01guration: When models are trained with images of objects in isolation, adding\n\ufb02ankers harms recognition. Adding two \ufb02ankers is the same or worse than adding just one and the\nsmaller the spacing between \ufb02anker and target, the more crowding occurs. These is because the\npooling operation merges nearby responses, such as the target and \ufb02ankers if they are close.\n\u2022 Similarity between target and \ufb02anker: Flankers more similar to targets cause more crowding,\n\u2022 Dependence on target location and contrast normalization:\n\nIn DCNNs and eccentricity-\ndependent models with contrast normalization, recognition accuracy is the same across all eccen-\ntricities. In eccentricity-dependent networks without contrast normalization, recognition does not\ndecrease despite presence of clutter when the target is at the center of the image.\n\u2022 Effect of pooling: adding pooling leads to better recognition accuracy of the models. Yet, in the\neccentricity model, pooling across the scales too early in the hierarchy leads to lower accuracy.\n\nbecause of the selectivity property of the learned DNN \ufb01lters.\n\nOur main conclusion is that when testing accuracy recognition of the target embedded in (place)\nimages, the eccentricity-dependent model \u2013 without contrast normalization and with spatial and scale\npooling at the end of the hierarchy \u2013 is robust to complex types of clutter, even though it had been\ntrained on images of objects in isolation. Yet, this occurs only when the target is at the center of the\nimage as it occurs when it is \ufb01xated by a human observer. Our analysis suggests that if we had access\nto a system for selecting target object location, such as the one proposed by [27], the eccentricity\ndependent model could be trained with lower sample complexity than other DCNN because it is\nrobust to some factors of image variation, such as clutter and scale changes. Translation invariance\nwould mostly be achieved through foveation.\n\n9\n\n\fAcknowledgments\n\nThis work was supported by the Center for Brains, Minds and Machines (CBMM), funded by NSF\nSTC award CCF - 1231216. A. Volokitin was also funded by Swiss Commission for Technology\nand Innovation (KTI, Grant No 2-69723-16), and thanks Luc Van Gool for his support. G. Roig was\npartly funded by SUTD SRG grant (SRG ISTD 2017 131). We also thank Xavier Boix, Francis Chen\nand Yena Han for helpful discussions.\n\nReferences\n\n[1] A. Krizhevsky, I. Sutskever, and G. E. Hinton, \u201cImagenet classi\ufb01cation with deep convolutional neural\nnetworks,\u201d in Advances in Neural Information Processing Systems 25 (F. Pereira, C. J. C. Burges, L. Bottou,\nand K. Q. Weinberger, eds.), pp. 1097\u20131105, Curran Associates, Inc., 2012.\n\n[2] K. Simonyan and A. Zisserman, \u201cVery deep convolutional networks for large-scale image recognition,\u201d\n\nCoRR, vol. abs/1409.1556, 2014.\n\n[3] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich,\n\n\u201cGoing deeper with convolutions,\u201d in Computer Vision and Pattern Recognition (CVPR), 2015.\n\n[4] K. He, X. Zhang, S. Ren, and J. Sun, \u201cDeep residual learning for image recognition,\u201d in Proceedings of the\n\nIEEE Conference on Computer Vision and Pattern Recognition, pp. 770\u2013778, 2016.\n\n[5] S. Ren, K. He, R. Girshick, and J. Sun, \u201cFaster r-cnn: Towards real-time object detection with region\n\nproposal networks,\u201d in Advances in neural information processing systems, pp. 91\u201399, 2015.\n\n[6] I. J. Goodfellow, J. Shlens, and C. Szegedy, \u201cExplaining and harnessing adversarial examples,\u201d arXiv\n\npreprint arXiv:1412.6572, 2014.\n\n[7] Y. Luo, X. Boix, G. Roig, T. A. Poggio, and Q. Zhao, \u201cFoveation-based mechanisms alleviate adversarial\n\nexamples,\u201d arXiv:1511.06292, 2015.\n\n[8] M. D. Zeiler and R. Fergus, \u201cVisualizing and understanding convolutional networks,\u201d in European confer-\n\nence on computer vision, pp. 818\u2013833, Springer, 2014.\n\n[9] D. Whitney and D. M. Levi, \u201cVisual crowding: A fundamental limit on conscious perception and object\n\nrecognition,\u201d Trends in cognitive sciences, vol. 15, no. 4, pp. 160\u2013168, 2011.\n\n[10] D. M. Levi, \u201cCrowding\u2014an essential bottleneck for object recognition: A mini-review,\u201d Vision research,\n\nvol. 48, no. 5, pp. 635\u2013654, 2008.\n\n[11] H. Bouma, \u201cInteraction effects in parafoveal letter recognition,\u201d Nature, vol. 226, pp. 177\u2013178, 1970.\n[12] F. L. Kooi, A. Toet, S. P. Tripathy, and D. M. Levi, \u201cThe effect of similarity and duration on spatial\n\ninteraction in peripheral vision,\u201d Spatial vision, vol. 8, no. 2, pp. 255\u2013279, 1994.\n\n[13] T. A. Nazir, \u201cEffects of lateral masking and spatial precueing on gap-resolution in central and peripheral\n\nvision,\u201d Vision research, vol. 32, no. 4, pp. 771\u2013777, 1992.\n\n[14] W. P. Banks, K. M. Bachrach, and D. W. Larson, \u201cThe asymmetry of lateral interference in visual letter\n\nidenti\ufb01cation,\u201d Perception & Psychophysics, vol. 22, no. 3, pp. 232\u2013240, 1977.\n\n[15] G. Francis, M. Manassi, and M. Herzog, \u201cCortical dynamics of perceptual grouping and segmentation:\n\nCrowding,\u201d Journal of Vision, vol. 16, no. 12, pp. 1114\u20131114, 2016.\n\n[16] J. Freeman and E. P. Simoncelli, \u201cMetamers of the ventral stream,\u201d Nature neuroscience, vol. 14, no. 9,\n\npp. 1195\u20131201, 2011.\n\n[17] B. Balas, L. Nakano, and R. Rosenholtz, \u201cA summary-statistic representation in peripheral vision explains\n\nvisual crowding,\u201d Journal of vision, vol. 9, no. 12, pp. 13\u201313, 2009.\n\n[18] T. Poggio, J. Mutch, and L. Isik, \u201cComputational role of eccentricity dependent cortical magni\ufb01cation,\u201d\n\narXiv preprint arXiv:1406.1770, 2014.\n\n[19] B. Cheung, E. Weiss, and B. A. Olshausen, \u201cEmergence of foveal image sampling from learning to attend\n\nin visual scenes,\u201d International Conference on Learning Representations, 2016.\n\n[20] Y. LeCun, C. Cortes, and C. J. Burges, \u201cThe mnist database of handwritten digits,\u201d 1998.\n[21] Y. Bulatov, \u201cnotMNIST dataset.\u201d http://yaroslavvb.blogspot.ch/2011/09/notmnist-dataset.\n\nhtml, 2011. Accessed: 2017-05-16.\n\n[22] B. M. Lake, R. Salakhutdinov, and J. B. Tenenbaum, \u201cHuman-level concept learning through probabilistic\n\nprogram induction,\u201d Science, vol. 350, no. 6266, pp. 1332\u20131338, 2015.\n\n[23] F. Chen, G. Roig, X. Isik, L. Boix, and T. Poggio, \u201cEccentricity dependent deep neural networks: Modeling\n\ninvariance in human vision,\u201d in AAAI Spring Symposium Series, Science of Intelligence, 2017.\n\n10\n\n\f[24] S. Keshvari and R. Rosenholtz, \u201cPooling of continuous features provides a unifying account of crowding,\u201d\n\nJournal of Vision, vol. 16, no. 3, pp. 39\u201339, 2016.\n\n[25] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, \u201cGradient-based learning applied to document recognition,\u201d\n\nProceedings of the IEEE, vol. 86, no. 11, pp. 2278\u20132324, 1998.\n\n[26] B. Zhou, A. Lapedriza, J. Xiao, A. Torralba, and A. Oliva, \u201cLearning deep features for scene recognition\n\nusing places database,\u201d in Advances in neural information processing systems, pp. 487\u2013495, 2014.\n\n[27] V. Mnih, N. Heess, A. Graves, et al., \u201cRecurrent models of visual attention,\u201d in Advances in neural\n\ninformation processing systems, pp. 2204\u20132212, 2014.\n\n11\n\n\f", "award": [], "sourceid": 2880, "authors": [{"given_name": "Anna", "family_name": "Volokitin", "institution": "ETH Zurich"}, {"given_name": "Gemma", "family_name": "Roig", "institution": "Massachusetts Institute of Technology"}, {"given_name": "Tomaso", "family_name": "Poggio", "institution": "MIT"}]}