{"title": "Putting An End to End-to-End: Gradient-Isolated Learning of Representations", "book": "Advances in Neural Information Processing Systems", "page_first": 3039, "page_last": 3051, "abstract": "We propose a novel deep learning method for local self-supervised representation learning that does not require labels nor end-to-end backpropagation but exploits the natural order in data instead. Inspired by the observation that biological neural networks appear to learn without backpropagating a global error signal, we split a deep neural network into a stack of gradient-isolated modules. Each module is trained to maximally preserve the information of its inputs using the InfoNCE bound from Oord et al [2018]. Despite this greedy training, we demonstrate that each module improves upon the output of its predecessor, and that the representations created by the top module yield highly competitive results on downstream classification tasks in the audio and visual domain. The proposal enables optimizing modules asynchronously, allowing large-scale distributed training of very deep neural networks on unlabelled datasets.", "full_text": "Putting An End to End-to-End:\n\nGradient-Isolated Learning of Representations\n\nSindy L\u00f6we\u2217\n\nPeter O\u2019Connor\n\nBastiaan S. Veeling\u2217\n\nAMLab\n\nUniversity of Amsterdam\n\nloewe.sindy@gmail.com, basveeling@gmail.com\n\nAbstract\n\nWe propose a novel deep learning method for local self-supervised representation\nlearning that does not require labels nor end-to-end backpropagation but exploits\nthe natural order in data instead. Inspired by the observation that biological neural\nnetworks appear to learn without backpropagating a global error signal, we split\na deep neural network into a stack of gradient-isolated modules. Each module\nis trained to maximally preserve the information of its inputs using the InfoNCE\nbound from Oord et al. [2018]. Despite this greedy training, we demonstrate that\neach module improves upon the output of its predecessor, and that the representa-\ntions created by the top module yield highly competitive results on downstream\nclassi\ufb01cation tasks in the audio and visual domain. The proposal enables optimiz-\ning modules asynchronously, allowing large-scale distributed training of very deep\nneural networks on unlabelled datasets.\n\n1\n\nIntroduction\n\nModern deep learning models are typically optimized using end-to-end backpropagation and a global,\nsupervised loss function. Although empirically proven to be highly successful [Krizhevsky et al.,\n2012, Szegedy et al., 2015], this approach is considered biologically implausible. For one, supervised\nlearning requires large labeled datasets to ensure generalization. In contrast, children can learn to\nrecognize a new category based on a handful of samples. Additionally, despite some evidence for\ntop-down connections in the brain, there does not appear to be a global objective that is optimized by\nbackpropagating error signals [Crick, 1989, Marblestone et al., 2016]. Instead, the biological brain is\nhighly modular and learns predominantly based on local information [Caporale and Dan, 2008].\n\nIn addition to lacking a natural counterpart, the supervised training of neural networks with end-to-end\nbackpropagation suffers from practical disadvantages as well. Supervised learning requires labeled\ninputs, which are expensive to obtain. As a result, it is not applicable to the majority of available data,\nand suffers from a higher risk of over\ufb01tting, as the number of parameters required for a deep model\noften exceeds the number of labeled datapoints at hand. At the same time, end-to-end backpropagation\ncreates a substantial memory overhead in a na\u00efve implementation, as the entire computational graph,\nincluding all parameters, activations and gradients, needs to \ufb01t in a processing unit\u2019s working memory.\nCurrent approaches to prevent this require either the recomputation of intermediate outputs [Salimans\nand Bulatov, 2017] or expensive reversible layers [Jacobsen et al., 2018]. This inhibits the application\nof deep learning models to high-dimensional input data that surpass current memory constraints. This\nproblem is perpetuated as end-to-end training does not allow for an exact way of asynchronously\noptimizing individual layers [Jaderberg et al., 2017]. In a globally optimized network, every layer\nneeds to wait for its predecessors to provide its inputs, as well as for its successors to provide gradients.\n\n\u2217equal contribution\n\n33rd Conference on Neural Information Processing Systems (NeurIPS 2019), Vancouver, Canada.\n\n\fDownstream\tLinear\tClassifier\n\nGradientBlock\n\nmodule\t3\n\nmodule\t2\n\nmodule\t1\n\ninput\n\nThe\tInfoNCE\tObjective\n\n1\n\npositive\nsample\n\nN\t-\t1\n\nnegative\nsamples\n\n\n\n\ue238\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n+\n\n\n\n...\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\u22121\n\n\u22121\n\n\u22121\n\n\n\n\n\n\n\n\n\n+\n\n\n\nFigure 1: The Greedy InfoMax Learning Approach. (Left) For the self-supervised learning of representations,\nwe stack a number of modules through which the input is forward-propagated in the usual way, but gradients do\nnot propagate backward. Instead, every module is trained greedily using a local loss. (Right) Every encoding\nmodule maps its inputs zm\u22121\nt , which is used as the input\nfor the following module. The InfoNCE objective is used for its greedy optimization. This loss is calculated by\ncontrasting the predictions of a module for its future representations zm\nj , which\nenforces each module to maximally preserve the information of its inputs. We optionally employ an additional\nautoregressive module gar, which is not depicted here.\n\nt+k against negative samples zm\n\n)) = zm\n\nat time-step t to gm\n\nenc(GradientBlock(zm\u22121\n\nt\n\nt\n\nThis forward and backward locking of the network caused by the backpropagation algorithm impedes\nthe ef\ufb01ciency of hardware accelerator design due to a lack of locality.\n\nIn this paper, we introduce a novel learning approach, Greedy InfoMax (GIM), that improves\nupon these problems. Drawing inspiration from biological constraints, we remove end-to-end\nbackpropagation by dividing a deep architecture into gradient-isolated modules that we train using a\ngreedy, self-supervised loss per module. Given unlabeled high-dimensional sequential or spatial data,\nwe encode it iteratively, module by module. By using a loss that enforces the individual modules to\nmaximally preserve the information of their inputs, we enable the stacked model to collectively create\ncompact representations that can be used for downstream tasks. Our contributions are as follows:1\n\n\u2022 The proposed Greedy InfoMax algorithm achieves strong performance on audio and image\n\nclassi\ufb01cation tasks despite greedy self-supervised training.\n\n\u2022 This enables asynchronous, decoupled training of neural networks, allowing for training\n\narbitrarily deep networks on larger-than-memory input data.\n\n\u2022 We show that mutual information maximization is especially suited for layer-by-layer greedy\n\noptimization, and argue that this reduces the problem of vanishing gradients.\n\n2 Background\n\nIn order to create compact representations from data that are useful for downstream tasks, we assume\nthat natural data exhibit so-called slow features [Wiskott and Sejnowski, 2002]. It is theorized that such\nfeatures are highly effective for downstream tasks such as object detection or speech recognition. To\nillustrate: a patch of a few milliseconds of raw speech utterances shares information with neighboring\npatches such as the speaker identity, emotion, and phonemes, while it does not necessarily share these\nwith random patches drawn from other utterances. Similarly, a small patch from a natural image\nshares many aspects with neighboring patches such as the depicted object or lighting conditions.\n\nRecent work [Hjelm et al., 2019, Oord et al., 2018] has proposed how we can exploit this to learn\nrepresentations that maximize the mutual information shared among neighbors. In this work, we\nfocus speci\ufb01cally on Contrastive Predictive Coding (CPC) [Oord et al., 2018]. This self-supervised\n\n1Our code is available at https://github.com/loeweX/Greedy_InfoMax.\n\n2\n\n\fend-to-end learning approach extracts useful representations from sequential inputs by maximizing\nthe mutual information between the extracted representations of temporally nearby patches.\n\nIn order to achieve this, CPC \ufb01rst processes the sequential input signal x using a deep encoding\nmodel genc(xt) = zt, and additionally produces a representation ct that aggregates the information\nof all patches up to time-step t using an autoregressive model gar(z0:t) = ct. Then, the mutual\ninformation between the extracted representations zt+k and ct of temporally nearby patches is\nmaximized by employing a speci\ufb01cally designed global probabilistic loss: Following the principles\nof Noise Contrastive Estimation (NCE) [Gutmann and Hyv\u00e4rinen, 2010], CPC takes a bag X =\n{zt+k, zj1 , zj2 , ...zjN \u22121 } for each delay k, with one \u201cpositive sample\u201d zt+k which is the encoding of\nthe input that follows k time-steps after ct, and N \u2212 1 \u201cnegative samples\u201d zjn which are uniformly\ndrawn from all available encoded input sequences.\n\nEach pair of encodings (zj, ct) is scored using a function f (\u00b7) to predict how likely it is that the\ngiven zj is the positive sample zt+k.\nIn practice, Oord et al. [2018] use a log-bilinear model\n\nscores from f (\u00b7) are used to predict which sample in the bag X is correct, leading to the InfoNCE loss:\n\nj Wkct(cid:1) with a unique weight-matrix Wk for each k-steps-ahead prediction. The\n\nfk(zj, ct) = exp(cid:0)zT\n\n(1)\n\nLN = \u2212Xk\n\nX\"log\n\nE\n\nfk(zt+k, ct)\n\nPzj \u2208X fk(zj, ct)# .\n\nThis loss is used to optimize both the encoding model genc and the auto-regressive model gar to\nextract the features that are consistent over neighboring patches but which diverge between random\npairs of patches. At the same time, the scoring model fk learns to use those features to correctly\nclassify the matching pair. In practice, the loss is trained using stochastic gradient descent with mini-\nbatches drawn from a large dataset of sequences, and negative samples drawn uniformly from all\nsequences in the minibatch. Note, that no min-max issues arise as found in adversarial training.\n\nAs a result of this con\ufb01guration, one can derive that the optimal solution for f is proportional to the\nfollowing density ratio [Oord et al., 2018]:\n\nfk(zt+k, ct) \u221d\n\np(zt+k|ct)\n\np(zt+k)\n\n.\n\n(2)\n\nThis insight allows us to reformulate \u2212LN as a lower bound on the mutual information I(zt+k, ct), as\ndemonstrated in the appendix of Oord et al. [2018] and proven by Poole et al. [2018]. Minimizing the\nloss LN thus optimizes the mutual information between consecutive patch representations I(zt+k, ct),\nwhich in itself lower bounds the mutual information I(xt+k, ct) between the future input xt+k and\nthe current representation ct. Hyvarinen and Morioka [2016] show that a similar patch-contrastive\nsetup leads to the extraction of a set of conditionally-independent components, such as Gabor-like\n\ufb01lters found in the early biological vision system.\n\nLayer-wise Information Preservation in Neuroscience Linsker [1988] developed the InfoMax\nprinciple in 1988. It theorizes that the brain learns to process its perceptions by maximally preserving\nthe information of the input activities in each layer. On top of this, neuroscience suggests that\nthe brain predicts its future inputs and learns by minimizing this prediction error [Friston, 2010].\nEmpirical evidence indicates, for example, that retinal cells carry signi\ufb01cant mutual information\nbetween the current and the future state of their own activity [Palmer et al., 2015]. Rao and Ballard\n[1999] indicate that this process may happen at each layer within the brain. Our proposal draws\nmotivation from these theories, resulting in a method that learns to preserve the information between\nthe input and the output of each layer by learning representations that are predictive of future inputs.\n\n3 Greedy InfoMax\n\nIn this paper, we pose the question if we can effectively optimize the mutual information between\nrepresentations at each layer of a model in isolation, enjoying the many practical bene\ufb01ts that greedy\ntraining (decoupled, isolated training of parts of a model) provides. In doing so, we introduce a novel\napproach for self-supervised representation learning: Greedy InfoMax (GIM). As depicted on the left\nside of Figure 1, we take a conventional deep learning architecture and divide it by depth into a stack\nof M modules. This decoupling can happen at the individual layer level or, for example, at the level\n\n3\n\n\fFigure 2: Groups of 4 image patches that excite a speci\ufb01c neuron, at 3 levels in the model (rows). Despite\nunsupervised greedy training, neurons appear to extract increasingly semantic features. Best viewed on screen.\n\nof blocks found in residual networks [He et al., 2016]. Rather than training this model end-to-end,\nwe prevent gradients from \ufb02owing between modules and employ a local self-supervised loss instead,\nadditionally reducing the issue of vanishing gradients.\n\nAs shown on the right side of Figure 1, each encoding module gm\nenc within our architecture maps\nthe output from the previous module zm\u22121\nenc(GradientBlock(zm\u22121\nt = gm\n)). No\ngradients are \ufb02owing between modules, which is enforced using a gradient blocking operator de\ufb01ned\nas GradientBlock(x) , x, \u2207 GradientBlock(x) , 0. Oord et al. [2018] propose to use the output\nof an autoregressive model gar(z0:t) = ct to contrast against future predictions zt+k. However, our\npreliminary results showed that this did not improve results if applied at every module in the stack and\noptimizing it requires backpropagation through time, which is considered biologically implausible.\nTherefore, we train each module gm\n\nenc using the following module-local InfoNCE loss:\n\nto an encoding zm\n\nt\n\nt\n\nk (zm\nf m\n\nt+k, zm\n\nT W m\n\nt+k\n\nt ) = exp(cid:16)zm\nN = \u2212Xk\n\nE\n\nLm\n\nX\"log\n\nk zm\n\nt (cid:17)\nPzm\n\nj\n\nf m\nk (zm\n\u2208X f m\n\nt+k, zm\nt )\nk (zm\nj , zm\n\nt )# .\n\nzM\nt = gM\n\nenc(cid:0)gM \u22121\n\nenc (cid:0)\u00b7 \u00b7 \u00b7 g1\n\nenc (xt)(cid:1)(cid:1) .\n\n(3)\n\n(4)\n\n(5)\n\nAfter convergence of all modules, the scoring functions f m\nfeed-forward neural network architecture that extracts features zM\nt\n\nk (\u00b7) can be discarded, leaving a conventional\n\nfor downstream tasks:\n\nFor certain downstream tasks, a broad context is essential. For example, in speech recognition, the\nreceptive \ufb01eld of zM\nt might not carry enough information to distinguish phonetic structures. To\nprovide this context, we reintroduce the autoregressive model gar as an independent module that we\noptionally append to the stack of encoding modules, resulting in a context-aggregate representation\ncM\nt = gM\nrole. We train this module independently using the following altered scoring function:\n\n(cid:1)(cid:1). In practice, a GRU or PixelCNN-style model can serve in this\nt ) = exp(cid:16)GradientBlock(cid:0)zM \u22121\nt+k (cid:1)T\n\nar(cid:0)GradientBlock(cid:0)zM \u22121\n\nIterative Mutual Information Maximization Similarly to the InfoNCE loss in Equation (1), our\nmodule-local InfoNCE loss in Equation (4) maximizes a lower bound on the mutual information\nI(zm\n\nt ) between nearby patch representations, encouraging the extraction of slow features.\n\nt (cid:17) .\n\nk (zM \u22121\nf M\n\nt+k , cM\n\nk cM\n\nt+k, zm\n\nW M\n\n(6)\n\n0:t\n\nMost importantly, it follows from Oord et al. [2018], that the module-local InfoNCE loss also\nmaximizes the lower bound of the mutual information I(zm\u22121\nt ) between the future input to a\nmodule and its current representation. This can be seen as a maximization of the mutual information\nbetween the input and the output of a module, subject to the constraint of temporal disparity. Thus,\nthe InfoNCE loss can successfully enforce each module to maximally preserve the information of\nits inputs, while providing the necessary regularization [Hu et al., 2017, Krause et al., 2010] for\ncircumventing degenerate solutions. These factors contribute to ensuring that the greedily optimized\nmodules provide meaningful inputs to their successors and that the network as a whole provides\nuseful features for downstream tasks without the use of a global error signal.\n\nt+k , zm\n\nPractical Bene\ufb01ts Applying GIM to high-dimensional inputs, we can optimize each module in\nsequence to decrease the memory costs during training. In the most memory-constrained scenario,\n\n4\n\n\fTable 1: STL-10 classi\ufb01cation results on the test set. The\nGIM model outperforms the CPC model, despite a lack of\nend-to-end backpropagation and without the use of a global\nobjective. (\u00b1 standard deviation over 4 training runs.)\n\nTable 2: GPU memory consumption during\ntraining. All models consist of the ResNet-50\narchitecture and only differ in their training ap-\nproach. GIM allows ef\ufb01cient greedy training.\n\nMethod\n\nAccuracy (%)\n\nMethod\n\nGPU memory (GB)\n\nDeep InfoMax [Hjelm et al., 2019]\nPredsim [N\u00f8kland and Eidnes, 2019]\n\nRandomly initialized\nSupervised\nGreedy Supervised\nCPC\n\nGreedy InfoMax (GIM)\n\n78.2\n80.8\n\n27.0\n71.4\n65.2\n\n80.5 \u00b1 3.1\n\n81.9 \u00b1 0.3\n\nSupervised\nCPC\n\nGIM - all modules\nGIM - 1st module\n\n6.3\n7.7\n\n7.0\n2.5\n\nindividual modules can be trained, frozen, and their outputs stored as a dataset for the next module,\nwhich effectively removes the depth of the network as a factor of the memory complexity.\n\nAdditionally, GIM allows for training models on larger-than-memory input data with architectures\nthat would otherwise exceed memory limitations. Leveraging the conventional pooling and strided\nlayers found in common network architectures, we can start with small patches of the input, greedily\ntrain the \ufb01rst module, extract the now compressed representation spanning larger windows of the\ninput and train the following module using these.\n\nLast but not least, GIM provides a highly \ufb02exible framework for the training of neural networks. It\nenables the training of individual parts of an architecture at varying update frequencies. When a\nhigher level of abstraction is needed, GIM allows for adding new modules on top at any moment of\nthe optimization process without having to \ufb01ne-tune previous results.\n\n4 Experiments\n\nWe test the applicability of the GIM approach to the visual and audio domain. In both settings, a\nfeature-extraction model is divided by depth into modules and trained without labels using GIM.\nThe representations created by the \ufb01nal (frozen) module are then used as the input for a linear\nclassi\ufb01er, whose accuracy scores provide us with a proxy for the quality and generalizability of the\nrepresentations created by the self-supervised model.\n\n4.1 Vision\n\nTo apply Greedy InfoMax to natural images, we impose a top-down ordering on 2D images. We\nfollow H\u00e9naff et al. [2019], Oord et al. [2018] by extracting a grid of partly-overlapping patches\nfrom the image to restrict the receptive \ufb01elds of the representations. For each patch xi,j in row i\nand column j of this grid, we predict up to K patches xi+K,j in the rows underneath, skipping the\n\ufb01rst overlapping patch xi+1,j . Random contrastive samples are drawn with replacement from all\nsamples available inside a batch, using 16 contrastive samples for each evaluation of the loss. No\nautoregressive module gar is used for GIM in this regime.\n\nExperimental Details We focus on the STL-10 dataset [Coates et al., 2011] which provides an\nadditional unlabeled training dataset. For data augmentation, we take random 64 \u00d7 64 crops from the\n96 \u00d7 96 images, \ufb02ip horizontally with probability 0.5 and convert to grayscale. We divide each image\nof 64 \u00d7 64 pixels into a total of 7 \u00d7 7 local patches, each of size 16 \u00d7 16 with 8 pixels overlap. The\npatches are encoded by a ResNet-50 v2 model [He et al., 2016] without batch normalization [Ioffe\nand Szegedy, 2015]. We split the model into three gradient-isolated modules that we train in sync and\nwith a constant learning rate. After convergence, a linear classi\ufb01er is trained \u2013 without \ufb01netuning the\nrepresentations \u2013 using a conventional softmax activation and cross-entropy loss. This linear classi\ufb01er\naccepts the patch representations zM\ni,j from the \ufb01nal module and \ufb01rst average-pools these, resulting in\na single vector representation zM . Remaining implementation details are presented in Appendix A.1.\n\n5\n\n\f(a) First Module\n\n(b) Second Module\n\n(c) Third Module\n\nFigure 3: Training curves for optimizing all modules simultaneously (blue) or iteratively, one at a time (red).\nWhile there is no difference in the training methods for the \ufb01rst module (a), later modules (b, c) start out with a\nlower loss and tend to over\ufb01t more when trained iteratively on top of already converged modules.\n\nResults As shown in Table 1, Greedy InfoMax (GIM) outperforms its end-to-end trained CPC\ncounterpart, despite its unsupervised features being optimized greedily without any backpropagation\nbetween modules. An equivalent randomly initialized feature extraction model exhibits poor perfor-\nmance, showing that GIM extracts useful features. Training the feature extraction model end-to-end\nand fully supervised performs worse, likely due to the small size of the annotated dataset resulting in\nover\ufb01tting. Although this could potentially be circumvented through regularization techniques [De-\nVries and Taylor, 2017], the self-supervised methods do not appear to require regularization as they\nbene\ufb01t from the full unlabeled dataset. Using a greedy supervised approach for training the feature\nmodel impedes performance, which suggests that mutual information maximization is unique in its\ndirect applicability to greedy optimization.\n\nIn comparison with the recently proposed Deep InfoMax model from Hjelm et al. [2019] which uses a\nslightly different end-to-end mutual information maximization approach, AlexNet [Krizhevsky et al.,\n2012] as their feature-extraction model and an additional hidden layer in the supervised classi\ufb01cation\nmodel, GIM comes out favorably. Finally, we see that we outperform the state-of-the-art biologically\ninspired Predsim model from N\u00f8kland and Eidnes [2019], which trains individual layers of a VGG\nlike architecture [Simonyan and Zisserman, 2014] using two supervised loss functions.\n\nIn Figure 2, we visualize patches that neurons in intermediate modules of the GIM model are sensitive\nto. This demonstrates that modules later in the model focus on increasingly abstract features. Overall,\nthe results demonstrate that complicated visual tasks can be approached using greedy self-supervised\noptimization, which can utilize large-scale unlabeled datasets.\n\nAsynchronous memory usage GIM provides a signi\ufb01cant practical advantage arising from the\ngreedy nature of optimization: modules can be trained in isolation given cached outputs from previous\nmodules, effectively removing the depth of the network as a factor of the memory complexity.\nMeasuring the allocated GPU memory of the previously studied models during training (Table 2),\nindicates that this theoretical bene\ufb01t holds in practice as well. After splitting the architecture into\nthree separately trainable modules, we can reduce the GPU memory consumption by a factor of 2.8 by\ntraining the modules asynchronously (GIM - 1st module) compared to training them simultaneously\n(GIM - all modules).\n\nWe evaluate whether training modules asynchronously in\ufb02uences the quality of the representations.\nFocusing on the extreme case, we optimize each module until convergence and \ufb01x its parameters,\nbefore we train the next module on top of it. This iteratively trained model achieves an accuracy\nof 79.8% on the image classi\ufb01cation downstream task. Thus, the performance declines slightly in\ncomparison to the simultaneously trained model, as previously shown in Table 1 with 81.9% accuracy.\n\nThe training curves of the two models as shown in Figure 3 provide some insight into this decreased\nperformance. The learning curves of the \ufb01rst module (Figure 3a) re\ufb02ect that there is no difference\nin its training in the two models. Modules two and three (Figures 3b and 3c), however, reveal a\ncrucial difference. The iteratively trained modules show a larger divergence between the training and\nvalidation loss, indicating stronger over\ufb01tting. We tentatively attribute this to the regularizing effect\nfrom the initially noisy inputs received by the higher modules when training simultaneously.\n\n6\n\n02004006008001000Epoch1.01.52.02.5InfoNCE Loss02004006008001000Epoch1.01.52.02.5InfoNCE Lossiterative trainiterative valsimultaneous trainsimultaneous val02004006008001000Epoch1.01.52.02.5InfoNCE Loss\fTable 3: Results for classifying speaker identity and phone labels in the LibriSpeech dataset. All models use the\nsame audio input sizes and the same architecture. Greedy InfoMax creates representations that are useful for\naudio classi\ufb01cation tasks despite its greedy training and lack of a global objective.\n\nMethod\n\nRandomly initialized b\nMFCC features b\nSupervised\nGreedy Supervised\nCPC [Oord et al., 2018] a\n\nGreedy InfoMax (GIM)\n\nPhone\n\nClassi\ufb01cation\nAccuracy (%)\n\nSpeaker\n\nClassi\ufb01cation\nAccuracy (%)\n\n27.6\n39.7\n77.7\n73.4\n64.9\n\n62.5\n\n1.9\n17.6\n98.9\n98.7\n99.6\n\n99.4\n\naIn the original implementation, Oord et al. [2018] achieved 64.6% for the phone and 97.4% for the speaker\n\nclassi\ufb01cation task. bBaseline results from Oord et al. [2018].\n\n4.2 Audio\n\nWe evaluate GIM in the audio domain on the sequence-global task of speaker classi\ufb01cation and the\nlocal task of phone classi\ufb01cation (distinct phonetic sounds that make up pronunciations of words).\nThese two tasks are interesting for self-supervised representation learning as the former requires\nrepresentations that discriminate speakers but are invariant to content, while the latter requires the\nopposite. Strong performance on both tasks thus suggests strong generalization and disentanglement.\n\nExperimental Details We follow the setup of Oord et al. [2018] unless speci\ufb01ed otherwise and use\na 100-hour subset of the publicly available LibriSpeech dataset [Panayotov et al., 2015]. It contains\nthe utterances of 251 different speakers with aligned phone labels divided into 41 classes. These phone\nlabels were provided by Oord et al. [2018] who obtained them by force-aligning phone sequences\nusing the Kaldi toolkit [Povey et al., 2011] and pre-trained models on Librispeech [Panayotov, 2014].\nWe \ufb01rst train the self-supervised model consisting of \ufb01ve convolutional layers and one autoregressive\nmodule, a single-layer gated recurrent unit (GRU). After convergence, a linear multi-class classi\ufb01er\nis trained on top of the context-aggregate representation cM without \ufb01ne-tuning the representations.\nRemaining implementation details are presented in Appendix A.2.\n\nResults Following Table 3, we analyze the performance of models on phone and speaker classi-\n\ufb01cation accuracy. Randomly initialized features perform poorly, demonstrating that both tasks re-\nquire complex representations. The traditional, hand-engineered MFCC features are commonly used\nin speech recognition systems [Ganchev et al., 2005], and improve over the random features, but\nprovide limited linear separability on both tasks. On the speaker classi\ufb01cation task, CPC and GIM\noutperform the supervised baselines despite their feature models having been trained without labels,\nand GIM without end-to-end backpropagation. In this setting, both GIM and Greedy Supervised,\nwhere individual layers are trained greedily with a supervised loss function, achieve similar results to\ntheir respective end-to-end trained counterparts (CPC and Supervised). When classifying phones,\nCPC does not reach the supervised performance (64.9% versus 77.7%). GIM achieves 62.5%, while\nGreedy Supervised accomplishes 73.4%. Thus, in contrast to the vision experiments (Section 4.1), we\nsee similar differences in performance between the greedily trained models (GIM and Greedy Super-\nvised) when compared to their respective end-to-end optimized counterparts (CPC and Supervised).\n\nOverall, the discrepancy between better-than-supervised performance on the speaker task and less-\nthan-optimal performance on the phone task suggests that GIM and CPC are biased towards extracting\nsequence-global features.\n\nAblation study The local greedy training enabled by GIM provides a step towards biologically\nplausible optimization and improves memory ef\ufb01ciency. However, the autoregressive module gar\naggregates its inputs over multiple patches and employs Backpropagation Through Time (BPTT),\nwhich puts a damper on both bene\ufb01ts. In Table 4, we present results on the performance of ablated\nmodels that restrict the \ufb02ow of gradients through time.\n\n7\n\n\fMethod\n\nAccuracy\n\n(%)\n\nSpeaker Classi\ufb01cation\nGreedy InfoMax (GIM)\nGIM without BPTT\nGIM without gar\n\nPhone Classi\ufb01cation\nGreedy InfoMax (GIM)\nGIM without BPTT\nGIM without gar\n\n99.4\n99.2\n99.1\n\n62.5\n55.5\n50.8\n\nTable 4: Ablation studies on the Lib-\nriSpeech dataset for removing the bi-\nologically implausible and memory-\nheavy backpropagation through time.\n\nFigure 4: Speaker Classi\ufb01cation error rates on a log scale (lower\nis better) for intermediate representations (layers 1 to 5), as well\nas for the \ufb01nal representation created by the autoregressive layer\n(corresponding to the results in Table 3).\n\nIn order to limit the \ufb02ow of gradients through time, we modify the autoregressive module. In\ngeneral, the autoregressive module gar takes the current input zt, as well as the hidden state of the\nprevious time-step ht\u22121, in order to produce its output ct, i.e. ct = gar(zt, ht\u22121) (omitting the\nmodule-index m here for brevity). In the standard GIM model, we block the \ufb02ow of gradients to\nthe previous module, such that ct = gar(GradientBlock(zt), ht\u22121). In the ablation GIM without\nBPTT, we remove BPTT by blocking the \ufb02ow of gradients between time-steps, such that ct =\ngar(GradientBlock(zt), GradientBlock(ht\u22121)). For the ablation GIM without gar, we remove the\nautoregressive module entirely. Here, the linear classi\ufb01er is applied to the representation created by\nthe last encoding module (i.e. zt).\n\nIn Table 4, we present the performance of the ablated models. Together, these two ablations indicate\na crucial difference between the tested downstream tasks. For the phone classi\ufb01cation task, we see a\nsteady decline of the performance when we reduce the modeling of temporal dependencies, indicating\ntheir importance for solving this task. When classifying the speaker identity, reducing the modeling\nof temporal dependencies in the ablated models barely in\ufb02uences their performance.\n\nTogether with the image classi\ufb01cation results from Section 4.1, where no autoregressive module was\nemployed either, this indicates that the GIM approach performs best on downstream tasks where\ntemporal or context dependencies do not need to be modeled by an autoregressive module. In these\nsettings, GIM can outperform the CPC model, which makes use of end-to-end backpropagation, a\nglobal objective, and BPTT.\n\nIntermediate module representations The greedy layer-wise training of GIM allows us to train\narbitrarily deep models without ever running into a memory constraint. We investigate how the\ncreated representations develop in each individual module by training a linear classi\ufb01er on top of each\nmodule and measuring their performance on the speaker classi\ufb01cation task. With results presented\nin Figure 4, we \ufb01rst observe that each GIM module improves upon the representations of their\npredecessor. Interestingly, CPC exhibits similar performance in intermediate modules despite these\nmodules relying solely on the error signal from the global loss function on the last module. This is\nin stark contrast with the supervised end-to-end model, whose intermediate layers lag behind their\ngreedily trained counterparts. This suggests that, in contrast to the supervised loss, the InfoMax\nprinciple \u201cstacks well\u201d, such that the greedy, iterative application of the InfoNCE loss performs\nsimilar to its global application.\n\n5 Related Work\n\nWe have studied the effectiveness of the self-supervised CPC approach [H\u00e9naff et al., 2019, Oord\net al., 2018] when applied to gradient-isolated modules, freeing the method from end-to-end back-\npropagation. There are a number of optimization algorithms that eliminate the need for backpropa-\ngation altogether [Balduzzi et al., 2015, Kohan et al., 2018, Lee et al., 2015, Lillicrap et al., 2016,\nOrorbia et al., 2018, Scellier and Bengio, 2017, Xiao et al., 2019]. In contrast to our method, these\n\n8\n\n123456Module0.010.050.10.51Error ratePretraining TypeGreedy InfoMaxCPCGreedy SupervisedSupervisedRandom init\fmethods employ a global supervised loss function and focus on \ufb01nding more biologically plausible\nways to assign credit to neurons.\n\nA recently published work by N\u00f8kland and Eidnes [2019] likewise demonstrates that backpropagation-\nfree layer-wise training is possible. Their similarity loss might be vaguely interpreted as another way\nof enforcing clustered representations. However, while our method achieves this entirely in a self-\nsupervised fashion by clustering temporally or spatially nearby inputs, their similarity loss groups\nrepresentations based on their class labels. Likewise, Belilovsky et al. [2019] showed that greedy\nlayer-wise training with a supervised loss can scale to ImageNet. In an attempt to validate information\nbottleneck theory, Elad et al. [2018] develop a supervised, layer-wise training method that maximizes\nthe mutual information between the outputs of a layer and the target whilst minimizing the mutual\ninformation between the inputs and outputs. In contrast to our proposal, these methods all rely on\nlabeled data.\n\nJaderberg et al. [2017] develop decoupled neural interfaces, which enjoy the same asynchronous\ntraining bene\ufb01ts as Greedy InfoMax (GIM), but achieve this by taking an end-to-end supervised loss\nand locally predicting its gradients. Bengio et al. [2007], Hinton et al. [2006] focus on deep belief\nnetworks and propose a greedy layer-wise unsupervised pretraining method based on Restricted\nBoltzmann Machine principles, followed by optimizing globally using a supervised loss. Lee et al.\n[2009] use convolutional deep belief networks for unsupervised pretraining on the TIMIT audio\ndataset and then evaluate their performance by training supervised classi\ufb01ers on top. Gao et al.\n[2018], Ver Steeg and Galstyan [2015] explore total correlation explanation, which is related to\nmutual information maximization, and show that it can be applied for layer-by-layer training.\n\nSeveral recent works investigated the utilization of mutual information maximization in a representa-\ntion learning setting [Belghazi et al., 2018, Hjelm et al., 2019, McAllester, 2018, Oord et al., 2018].\nPoole et al. [2018] analyse these recent works under a common framework and highlight that InfoNCE\nexhibits low variance at a cost of high bias and propose new lower bounds that allow for balancing\nthis bias/variance trade-off. However, the analysis of these improved bounds in the context of inter-\npatch mutual information optimization remains in order, and thus we focus on the original CPC In-\nfoNCE loss to bias the learned representations towards slow features [Wiskott and Sejnowski, 2002].\n\nOutside the information-theoretic framework, context prediction methods have been explored for\nunsupervised representation learning. A prominent approach in language processing is Word2Vec\n[Mikolov et al., 2013], in which a word is directly predicted given its context (continuous skip-gram).\nLikewise, Doersch et al. [2015] study such an approach for the visual domain. Similarly, graph neural\nnetworks use contrastive principles to learn unsupervised node embeddings based on their neighbors\n[Kipf and Welling, 2016, Nickel et al., 2011, 2015, Perozzi et al., 2014, Veli\u02c7ckovi\u00b4c et al., 2018].\nNoise contrastive estimation has also been explored for independent component analysis [Hyvarinen\nand Morioka, 2016, 2017, Hyvarinen et al., 2018]. Schmidhuber [1992] proposes a method where\nindividual features are minimized such that they cannot be predicted from other features, forcing\nthem to extract independent factors that carry statistical information, at the risk of neurons latching\nonto local independent noise sources in the input.\n\n6 Conclusion\n\nWe presented Greedy InfoMax, a novel self-supervised greedy learning approach. The relatively\nstrong performance demonstrates that deep neural networks do not necessarily require end-to-end\nbackpropagation of a supervised loss on perceptual tasks. Our proposal enables greedy self-supervised\ntraining, which makes the model less vulnerable to over\ufb01tting, reduces the vanishing gradient problem\nand enables memory-ef\ufb01cient asynchronous distributed training. While the biological plausibility\nof our proposal is limited by the use of negative samples and within-module backpropagation, the\nresults provide evidence that the theorized self-organization in biological perceptual networks is at\nleast feasible and effective in arti\ufb01cial networks, providing food for thought on the credit assignment\ndiscussion in perceptual networks [Bengio et al., 2015, Linsker, 1988].\n\nAcknowledgments\n\nWe thank Jorn Peters, Marco Federici, Rudy Corona, Pascal Esser, Joop Pascha and the anonymous\nreviewers for their insightful comments. This research was supported by Philips Research and the\nNVIDIA GPU Grant.\n\n9\n\n\fReferences\n\nDavid Balduzzi, Hastagiri Vanchinathan, and Joachim Buhmann. Kickback cuts backprop\u2019s red-tape:\nbiologically plausible credit assignment in neural networks. In Twenty-Ninth AAAI Conference\non Arti\ufb01cial Intelligence, 2015.\n\nMohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeshwar, Sherjil Ozair, Yoshua Bengio, Devon\nHjelm, and Aaron Courville. Mutual information neural estimation. In International Conference\non Machine Learning, pages 530\u2013539, 2018.\n\nEugene Belilovsky, Michael Eickenberg, and Edouard Oyallon. Greedy layerwise learning can scale\n\nto imagenet. In International Conference on Machine Learning, pages 583\u2013593, 2019.\n\nYoshua Bengio, Pascal Lamblin, Dan Popovici, and Hugo Larochelle. Greedy layer-wise training of\n\ndeep networks. In Advances in neural information processing systems, pages 153\u2013160, 2007.\n\nYoshua Bengio, Dong-Hyun Lee, Jorg Bornschein, Thomas Mesnard, and Zhouhan Lin. Towards\n\nbiologically plausible deep learning. arXiv preprint arXiv:1502.04156, 2015.\n\nNatalia Caporale and Yang Dan. Spike timing\u2013dependent plasticity: a hebbian learning rule. Annual\n\nReview of Neuroscience, 31:25\u201346, 2008.\n\nAdam Coates, Andrew Ng, and Honglak Lee. An analysis of single-layer networks in unsupervised\nfeature learning. In Proceedings of the fourteenth international conference on arti\ufb01cial intelligence\nand statistics, pages 215\u2013223, 2011.\n\nFrancis Crick. The recent excitement about neural networks. Nature, 337(6203):129\u2013132, 1989.\n\nTerrance DeVries and Graham W Taylor. Improved regularization of convolutional neural networks\n\nwith cutout. arXiv preprint arXiv:1708.04552, 2017.\n\nCarl Doersch, Abhinav Gupta, and Alexei A Efros. Unsupervised visual representation learning\nby context prediction. In Proceedings of the IEEE International Conference on Computer Vision,\npages 1422\u20131430, 2015.\n\nAdar Elad, Doron Haviv, Yochai Blau, and Tomer Michaeli. The effectiveness of layer-by-layer\n\ntraining using the information bottleneck principle. OpenReview, 2018.\n\nKarl Friston. The free-energy principle: a uni\ufb01ed brain theory? Nature reviews neuroscience, 11(2):\n\n127, 2010.\n\nTodor Ganchev, Nikos Fakotakis, and George Kokkinakis. Comparative evaluation of various mfcc\nimplementations on the speaker veri\ufb01cation task. In Proceedings of the SPECOM, volume 1, pages\n191\u2013194, 2005.\n\nShuyang Gao, Rob Brekelmans, Greg Ver Steeg, and Aram Galstyan. Auto-encoding total correlation\n\nexplanation. arXiv preprint arXiv:1802.05822, 2018.\n\nMichael Gutmann and Aapo Hyv\u00e4rinen. Noise-contrastive estimation: A new estimation principle\nfor unnormalized statistical models. In Proceedings of the Thirteenth International Conference on\nArti\ufb01cial Intelligence and Statistics, pages 297\u2013304, 2010.\n\nKaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual\n\nnetworks. In European conference on computer vision, pages 630\u2013645. Springer, 2016.\n\nOlivier J H\u00e9naff, Ali Razavi, Carl Doersch, SM Eslami, and Aaron van den Oord. Data-ef\ufb01cient\n\nimage recognition with contrastive predictive coding. arXiv preprint arXiv:1905.09272, 2019.\n\nGeoffrey E Hinton, Simon Osindero, and Yee-Whye Teh. A fast learning algorithm for deep belief\n\nnets. Neural computation, 18(7):1527\u20131554, 2006.\n\nR Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon, Karan Grewal, Adam Trischler, and\nYoshua Bengio. Learning deep representations by mutual information estimation and maximization.\nProceedings of the 7th International Conference on Learning Representations, 2019.\n\n10\n\n\fWeihua Hu, Takeru Miyato, Seiya Tokui, Eiichi Matsumoto, and Masashi Sugiyama. Learning discrete\nrepresentations via information maximizing self-augmented training. In Proceedings of the 34th\nInternational Conference on Machine Learning-Volume 70, pages 1558\u20131567. JMLR. org, 2017.\n\nAapo Hyvarinen and Hiroshi Morioka. Unsupervised feature extraction by time-contrastive learning\nand nonlinear ica. In Advances in Neural Information Processing Systems, pages 3765\u20133773,\n2016.\n\nAapo Hyvarinen and Hiroshi Morioka. Nonlinear ICA of Temporally Dependent Stationary Sources.\nIn Aarti Singh and Jerry Zhu, editors, Proceedings of the 20th International Conference on\nArti\ufb01cial Intelligence and Statistics, volume 54 of Proceedings of Machine Learning Research,\npages 460\u2013469, Fort Lauderdale, FL, USA, 2017. PMLR.\n\nAapo Hyvarinen, Hiroaki Sasaki, and Richard E Turner. Nonlinear ICA using auxiliary variables and\n\ngeneralized contrastive learning. May 2018.\n\nSergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by\n\nreducing internal covariate shift. arXiv preprint arXiv:1502.03167, 2015.\n\nJ\u00f6rn-Henrik Jacobsen, Arnold Smeulders, and Edouard Oyallon. i-revnet: Deep invertible networks.\n\narXiv preprint arXiv:1802.07088, 2018.\n\nMax Jaderberg, Wojciech Marian Czarnecki, Simon Osindero, Oriol Vinyals, Alex Graves, David\nSilver, and Koray Kavukcuoglu. Decoupled neural interfaces using synthetic gradients.\nIn\nProceedings of the 34th International Conference on Machine Learning-Volume 70, pages 1627\u2013\n1635, 2017.\n\nDiederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint\n\narXiv:1412.6980, 2014.\n\nThomas N Kipf and Max Welling. Variational graph auto-encoders. In NIPS Workshop on Bayesian\n\nDeep Learning, 2016.\n\nAdam A Kohan, Edward A Rietman, and Hava T Siegelmann. Error forward-propagation: Reusing\nfeedforward connections to propagate errors in deep learning. arXiv preprint arXiv:1808.03357,\n2018.\n\nAndreas Krause, Pietro Perona, and Ryan G Gomes. Discriminative clustering by regularized\ninformation maximization. In Advances in neural information processing systems, pages 775\u2013783,\n2010.\n\nAlex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classi\ufb01cation with deep convolu-\ntional neural networks. In Advances in neural information processing systems, pages 1097\u20131105,\n2012.\n\nDong-Hyun Lee, Saizheng Zhang, Asja Fischer, and Yoshua Bengio. Difference target propagation.\nIn Joint european conference on machine learning and knowledge discovery in databases, pages\n498\u2013515. Springer, 2015.\n\nHonglak Lee, Peter Pham, Yan Largman, and Andrew Y Ng. Unsupervised feature learning for\naudio classi\ufb01cation using convolutional deep belief networks. In Advances in neural information\nprocessing systems, pages 1096\u20131104, 2009.\n\nTimothy P Lillicrap, Daniel Cownden, Douglas B Tweed, and Colin J Akerman. Random synaptic\nfeedback weights support error backpropagation for deep learning. Nature communications, 7:\n13276, 2016.\n\nRalph Linsker. Self-organization in a perceptual network. Computer, 21(3):105\u2013117, 1988.\n\nAdam H Marblestone, Greg Wayne, and Konrad P Kording. Toward an integration of deep learning\n\nand neuroscience. Frontiers in computational neuroscience, 10:94, 2016.\n\nDavid McAllester. Information theoretic co-training. arXiv preprint arXiv:1802.07572, 2018.\n\n11\n\n\fTomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed representations\nof words and phrases and their compositionality. In Advances in neural information processing\nsystems, pages 3111\u20133119, 2013.\n\nMaximilian Nickel, Volker Tresp, and Hans-Peter Kriegel. A three-way model for collective learning\non multi-relational data. In Proceedings of the International Conference on Machine Learning,\nvolume 11, pages 809\u2013816, 2011.\n\nMaximilian Nickel, Kevin Murphy, Volker Tresp, and Evgeniy Gabrilovich. A review of relational\n\nmachine learning for knowledge graphs. Proceedings of the IEEE, 104(1):11\u201333, 2015.\n\nArild N\u00f8kland and Lars Hiller Eidnes. Training neural networks with local error signals.\n\nIn\n\nProceedings of the 36th International Conference on Machine Learning, 2019.\n\nAaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive\n\ncoding. arXiv preprint arXiv:1807.03748, 2018.\n\nAlexander G Ororbia, Ankur Mali, Daniel Kifer, and C Lee Giles. Conducting credit assignment by\n\naligning local representations. arXiv preprint arXiv:1803.01834, 2018.\n\nStephanie E Palmer, Olivier Marre, Michael J Berry, and William Bialek. Predictive information in a\nsensory population. Proceedings of the National Academy of Sciences, 112(22):6908\u20136913, 2015.\n\nVassil Panayotov. Kaldi pretrained model on LibriSpeech SAT and DNN. http://www.kaldi-asr.\n[Online; accessed 29-July-\n\norg/downloads/build/6/trunk/egs/librispeech/, 2014.\n2019].\n\nVassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: an asr corpus\nIn 2015 IEEE International Conference on Acoustics,\n\nbased on public domain audio books.\nSpeech and Signal Processing (ICASSP), pages 5206\u20135210. IEEE, 2015.\n\nAdam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito,\nZeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in\npytorch. 2017.\n\nBryan Perozzi, Rami Al-Rfou, and Steven Skiena. Deepwalk: Online learning of social repre-\nsentations. In Proceedings of the 20th ACM SIGKDD international conference on Knowledge\ndiscovery and data mining, pages 701\u2013710. ACM, 2014.\n\nBen Poole, Sherjil Ozair, A\u00e4ron van den Oord, Alexander A Alemi, and George Tucker. On variational\n\nlower bounds of mutual information. In NeurIPS Workshop on Bayesian Deep Learning, 2018.\n\nDaniel Povey, Arnab Ghoshal, Gilles Boulianne, Lukas Burget, Ondrej Glembek, Nagendra Goel,\nMirko Hannemann, Petr Motlicek, Yanmin Qian, Petr Schwarz, et al. The kaldi speech recognition\ntoolkit. In IEEE 2011 workshop on automatic speech recognition and understanding. IEEE Signal\nProcessing Society, 2011.\n\nRajesh PN Rao and Dana H Ballard. Predictive coding in the visual cortex: a functional interpretation\n\nof some extra-classical receptive-\ufb01eld effects. Nature neuroscience, 2(1):79, 1999.\n\nTim Salimans and Yaroslav Bulatov. Gradient checkpointing, 2017.\n\nBenjamin Scellier and Yoshua Bengio. Equilibrium propagation: bridging the gap between energy-\n\nbased models and backpropagation. Frontiers in computational neuroscience, 11:24, 2017.\n\nJ\u00fcrgen Schmidhuber. Learning factorial codes by predictability minimization. Neural Computation,\n\n4(6):863\u2013879, 1992.\n\nKaren Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image\n\nrecognition. arXiv preprint arXiv:1409.1556, 2014.\n\nChristian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Du-\nmitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In\nProceedings of the IEEE conference on computer vision and pattern recognition, pages 1\u20139, 2015.\n\n12\n\n\fPetar Veli\u02c7ckovi\u00b4c, William Fedus, William L Hamilton, Pietro Li\u00f2, Yoshua Bengio, and R Devon\n\nHjelm. Deep graph infomax. arXiv preprint arXiv:1809.10341, 2018.\n\nGreg Ver Steeg and Aram Galstyan. Maximally informative hierarchical representations of High-\nDimensional data. In Arti\ufb01cial Intelligence and Statistics, pages 1004\u20131012. jmlr.org, February\n2015.\n\nLaurenz Wiskott and Terrence J Sejnowski. Slow feature analysis: unsupervised learning of invari-\n\nances. Neural Comput., 14(4):715\u2013770, April 2002.\n\nWill Xiao, Honglin Chen, Qianli Liao, and Tomaso Poggio. Biologically-plausible learning algo-\nrithms can scale to large datasets. Proceedings of the 7th International Conference on Learning\nRepresentations, 2019.\n\n13\n\n\f", "award": [], "sourceid": 1730, "authors": [{"given_name": "Sindy", "family_name": "L\u00f6we", "institution": "University of Amsterdam"}, {"given_name": "Peter", "family_name": "O'Connor", "institution": "Brain Corporation"}, {"given_name": "Bastiaan", "family_name": "Veeling", "institution": "AMLab - University of Amsterdam"}]}