{"title": "Stimulus domain transfer in recurrent models for large scale cortical population prediction on video", "book": "Advances in Neural Information Processing Systems", "page_first": 7199, "page_last": 7210, "abstract": "To better understand the representations in visual cortex, we need to generate better predictions of neural activity in awake animals presented with their ecological input: natural video. Despite recent advances in models for static images, models for predicting responses to natural video are scarce and standard linear-nonlinear models perform poorly. We developed a new deep recurrent network architecture that predicts inferred spiking activity of thousands of mouse V1 neurons simultaneously recorded with two-photon microscopy, while accounting for confounding factors such as the animal's gaze position and brain state changes related to running state and pupil dilation. Powerful system identification models provide an opportunity to gain insight into cortical functions through in silico experiments that can subsequently be tested in the brain. However, in many cases this approach requires that the model is able to generalize to stimulus statistics that it was not trained on, such as band-limited noise and other parameterized stimuli. We investigated these domain transfer properties in our model and find that our model trained on natural images is able to correctly predict the orientation tuning of neurons in responses to artificial noise stimuli. Finally, we show that we can fully generalize from movies to noise and maintain high predictive performance on both stimulus domains by fine-tuning only the final layer's weights on a network otherwise trained on natural movies. The converse, however, is not true.", "full_text": "Stimulus domain transfer in recurrent models for\nlarge scale cortical population prediction on video\n\nFabian H. Sinz,1-2,5,7,* Alexander S. Ecker,2,4-6 Paul G. Fahey,1-2 Edgar Y. Walker,1-2\n\nErick Cobos, 1-2 Emmanouil Froudarakis,1-2 Dimitri Yatsenko,1-2\n\nXaq Pitkow,1-3 Jacob Reimer,1-2 Andreas S. Tolias1-3,5\n\n1 Department of Neuroscience, Baylor College of Medicine, Houston, TX, USA\n\n2 Center for Neuroscience and Arti\ufb01cial Intelligence, Baylor College of Medicine, Houston, TX, USA\n\n3 Department of Electrical and Computer Engineering, Rice University, Houston, TX, USA\n\n4 Centre for Integrative Neuroscience, University of T\u00fcbingen, Germany\n\n5 Bernstein Center for Computational Neuroscience, University of T\u00fcbingen, Germany\n\n6 Institute for Theoretical Physics, University of T\u00fcbingen, Germany\n7 Institute for Computer Science, University of T\u00fcbingen, Germany\n\n*sinz@bcm.edu\n\nAbstract\n\nTo better understand the representations in visual cortex, we need to generate better\npredictions of neural activity in awake animals presented with their ecological\ninput: natural video. Despite recent advances in models for static images, models\nfor predicting responses to natural video are scarce and standard linear-nonlinear\nmodels perform poorly. We developed a new deep recurrent network architecture\nthat predicts inferred spiking activity of thousands of mouse V1 neurons simulta-\nneously recorded with two-photon microscopy, while accounting for confounding\nfactors such as the animal\u2019s gaze position and brain state changes related to running\nstate and pupil dilation. Powerful system identi\ufb01cation models provide an opportu-\nnity to gain insight into cortical functions through in silico experiments that can\nsubsequently be tested in the brain. However, in many cases this approach requires\nthat the model is able to generalize to stimulus statistics that it was not trained on,\nsuch as band-limited noise and other parameterized stimuli. We investigated these\ndomain transfer properties in our model and \ufb01nd that our model trained on natural\nimages is able to correctly predict the orientation tuning of neurons in responses to\narti\ufb01cial noise stimuli. Finally, we show that we can fully generalize from movies\nto noise and maintain high predictive performance on both stimulus domains by\n\ufb01ne-tuning only the \ufb01nal layer\u2019s weights on a network otherwise trained on natural\nmovies. The converse, however, is not true.\n\n1\n\nIntroduction\n\nThe visual cortex represents natural stimuli in a complex and highly nonlinear way [1, 2]. In order to\nunderstand these representations, we need predictive models that can account for neural responses to\nnatural movies. This task is particularly challenging because a substantial portion of the response\nvariability in cortical neurons is not driven by the stimulus, but by other factors such as eye movements\nunder free-viewing conditions and brain state changes [3\u20139]. While deep convolutional networks have\nrecently been shown to improve prediction performance over linear-nonlinear type models [10\u201313]\nand are currently considered state-of-the-art, in V1 they have only been used to predict responses to\n\n32nd Conference on Neural Information Processing Systems (NeurIPS 2018), Montr\u00e9al, Canada.\n\n\fFigure 1: One timestep of the recurrent archi-\ntecture: The network consists of a core, a read-\nout, a shifter, and a modulator. The core uses\na CNN to generate a non-linear feature repre-\nsentation of one frame of the movie, which is\nthen fed into a GRU. The readout decomposes\nthe hidden state of the GRU into different scales\nand reads from one spatial location per neuron\nacross all features and scales using a spatial\ntransformer grid point. The shifter predicts a\nshift for the entire neuron population based on\nthe pupil position. The modulator predicts a\ngain term per time point and neuron based on\nthe running state and pupil dilation of the ani-\nmal.\n\nstatic natural images. These models also make sub-optimal use of the data, because they currently\ndo not account for stimulus-independent variability. Furthermore, system identi\ufb01cation is only one\nstep towards a better understanding of cortical representations. Successful models must be able to\ngeneralize beyond the stimulus statistics they are trained on to generate new insights that can be tested\nback in the brain. This domain transfer is a known hard problem in neural system identi\ufb01cation and\nmachine learning in general, since it requires generalization beyond the statistics of the training set.\nWe make three contributions towards this goal: (i) we propose a novel recurrent neural network\narchitecture that can simultaneously predict the responses of thousands of cortical neurons while\naccounting for neural variability caused by eye movements or brain state changes related to measurable\nfactors such as running state or pupil dilations; (ii) we demonstrate that training our model on natural\nmovies allows some extent of domain transfer and recovers neurons\u2019 tuning properties, such as\norientation tuning, direction tuning, and receptive \ufb01eld structure, under arti\ufb01cial stimuli; and (iii)\nwe analyze the limits of domain transfer to show that models trained directly on the target domain\nalways outperform those trained on other domains in terms of predictive performance. However,\nwe also demonstrate that the nonlinear feature representation learned on natural movies can transfer\nwell to the noise domain by \ufb01ne-tuning the last layer\u2019s weights, while the converse is not true (i. e.\ngeneralization from noise to movies). In fact, there exists a single set of feature weights that leads to\noptimal performance in both domains, even though we currently cannot identify this set of weights\nfrom natural movies alone.\n\n2 Network Architecture\n\nOur network consists of four components (Figure 1): a core providing nonlinear recurrent features\nfrom the video frames, a readout mapping the core features to each neuron\u2019s activity, a shifter\npredicting receptive \ufb01eld shifts from pupil position, and a modulator providing a gain factor for each\nneuron depending on the running state and the pupil dilation of the animal (Figure 1).\n\nCore The core consists of a three-layer, 2d convolutional neural network (CNN), applied separately\nto each frame, followed by a convolutional Gated Recurrent Unit (GRU) [14]. Each CNN layer\nis composed of a 2d convolution layer, a batch normalization layer [15], and an ELU nonlinearity\n[16]. The \ufb01rst layer has a skip connection into the third layer which is stacked onto the input. That\nis, if the \ufb01rst and second layer have k1 and k2 output channels, the third layer has k1 + k2 input\nchannels. Finally, similar to DenseNets [17], the outputs of all layers are stacked and fed to a\none-layer convolutional GRU that models the lag between input and neural response and recurrently\ncomputes nonlinear features. Previous work demonstrated that neural response properties like\ndirection selectivity can be modeled by recurrent networks [18].\n\n2\n\n\u2206xShifter NetworkModulator Network Stacked CNNFeature ExtractorGRUFrame t+1Frame t-1Frame tReadoutReadoutCore\fijk wijkvijk [10] or a factorized layer(cid:80)\n\napproach of using a fully connected layer(cid:80)\n\nReadout We model the neural response as an instantaneous af\ufb01ne function of the hidden state of\nthe GRU at time t followed by an ELU nonlinearity and an offset of 1 to make the response positive.\nAt each point in time the hidden state of the GRU is a tensor v \u2208 Rw\u00d7h\u00d7c. The straightforward\nijk wijwkvijk\nover features and space [12] requires strong regularization to deal with the large number of parameters\nand can potentially lead to ghosting artifacts where the receptive \ufb01eld of a neuron shows up at several\nlocations if the spatial components wij are non-zero at more than one spatial location because of\n\ufb01tting noise or ambiguities in the data caused by gaze shifts. To circumvent these issues we explicitly\nincorporate the prior knowledge that a neuron only reads out from one particular location in space\nand model each neuron with a spatial transformer layer [19] with a single grid point reading from\nvijk. Thus, each neuron i is parameterized by a relative spatial location (xi, yi) \u2208 [\u22121, 1]2 at which\nthe spatial transformer layer extracts a local feature vector vxiyi: by bilinearly interpolating the\nadjacent pixels; a linear-nonlinear function later combines these features into a single neural response\nyi = f (w(cid:62)\ni vxiyi: + bi). Since, a priori, we do not know the spatial location of the neuron, the\n(xi, yi) become part of the network parameters and are learned via gradient descent. However, this\nintroduces another problem: when the initial estimate of the location is far away from the neuron\u2019s\nactual location, there is little gradient information to nudge the grid point to the correct place. We\ntherefore decompose vijk into (cid:96) spatial scales through repeated application of a p\u00d7 p average pooling\nlayer with stride p until the smaller spatial dimension is only one pixel in size: v(j) = pool(j)(v).\nThe spatial transformer layer then extracts (cid:96) feature vectors from the same relative location (xi, yi)\nat each scale and stacks them into a single feature vector of dimension k \u00d7 (cid:96) fed to the \ufb01nal af\ufb01ne\nxiyi: + bi) (Figure 1). Importantly, the relative spatial\n\nj=1 w(cid:62)\n\njiv(j)\n\nfunction and nonlinearity yi = f ((cid:80)(cid:96)\n\nlocation is shared across scales.\n\nShifter Unlike primates, mice are not trained to \ufb01xate their gaze in a single position, complicating\neye tracking. To model the responses of thousands of neurons in a free viewing experiment, we take\nan alternative approach and directly estimate a receptive \ufb01eld shift for all neurons from the tracked\npupil position solely based on optimizing the predictive performance of the network. Speci\ufb01cally,\nwe feed the pupil location p \u2208 R2 into a network that predicts a shift \u2206x = (\u2206x, \u2206y) \u2208 R2 for\neach time point which is added onto all locations xi = (xi, yi) of the spatial transformer readout.\nNote that the pupil location is measured in coordinates of the camera recording the eye, while the\nshift needs to be applied in monitor coordinates. This transformation can either be estimated by a\ncalibration procedure [20\u201322], or learned from the data using regression on pairs of eye camera\u2013\nmonitor coordinates. Our approach differs from previous ones in that it estimates gaze shifts purely\nbased on prediction performance. To a \ufb01rst approximation, the mapping from pupil coordinates to\ngaze shifts is af\ufb01ne. We therefore use a one layer perceptron (MLP) with a tanh nonlinearity for\npredicting \u2206x for all neurons. We empirically found that clipping xi + \u2206x back to [\u22121, 1]2 improves\nthe performance of the network.\n\nModulator To account for \ufb02uctuations in neural responses unrelated to the visual stimulus, we\nuse variables known to correlate with brain state\u2014pupil dilations (and their derivative) and absolute\nrunning speed of the animal [4, 5, 23]\u2014to predict, per timepoint, a neuron-speci\ufb01c multiplicative\ngain factor applied to the output of the readout layer. We use a GRU followed by a fully connected\nlayer and an exponential nonlinearity offset by one to predict this factor and model the unknown\ndelay between behavioral state and neural gain.\n\n3 Related Work\n\nThere is a number of previous studies that predict neural responses to natural images or video,\ndiffering in the degree to which parts of the network are hand-crafted, the complexity of the network,\nthe neural responses they are \ufb01tted to (electrophysiology vs. two-photon), the species the data was\nrecorded in (mouse, cat, monkey), whether the animal was anaesthetized, and whether multiple\nneurons share parts of the network. None of the previous approaches predict a comparably large\nnumber of neurons, very few use video, and none simultaneously account for eye shifts and brain\nstate modulations.\nGallant and colleagues were the \ufb01rst to \ufb01t models to movies of natural scenes predicting the responses\nin macaque area V1 and MT [1, 24\u201326]; their models are either spatio-temporal linear-nonlinear\n\n3\n\n\fmodels or use hand-crafted non-linear features such as power in the Fourier domain or divisive\nnormalization. Since the monkey is \ufb01xating during their recordings, there is less need to consider\neye movement and brain state. Lau et al. trained a multi-layer perceptron with inputs from different\ndelayed time points to predict responses of V1 neurons to random bars [27]. Vintch et al. trained\na two state linear-nonlinear model with a convolutional \ufb01rst layer to predict cell responses in\nmonkey V1 [28]. Similar to our work, Batty et al. used a multi-layer recurrent network as a feature\nrepresentation for a linear-nonlinear model to predict retinal ganglion cells [29]. Sussillo et al. used a\nvariational autoencoder to infer latent low dimensional dynamics to explain neural responses [30];\ntheir model was only tested on synthetically generated data. Other studies predict neural responses to\nstatic arti\ufb01cial or natural images [10\u201312, 26, 27, 31, 32]. Zipser and Andersen were one of the \ufb01rst\nto use neural networks to predict neural data; they used visual input and eye position to model the\nresponses of neurons in area 7a whose neurons are involved in visuo-motor coordination [33].\n\n4 Experiments\n\nNeural and Behavioral Data Our data consists of three sets of 1344-4692 simultaneously recorded\ndeconvolved \ufb02uorescence traces [34] from two-photon scans in mouse visual cortex area V1 L2/3,\ncollected from three animals using a large-\ufb01eld-of-view mesoscope [35]. Cells were selected based\non spatial features of the segmented masks, but disregarding visual responsiveness. The acquisition\nframe rate was roughly 6Hz. Pupil position, dilation, and absolute running speed of the animal were\nmonitored with an eye tracking camera and a styrofoam treadmill. The contour of the pupil was\nextracted semi-automatically for each frame. The center and the major radius of a \ufb01tted ellipse were\nused as the position and the dilation of the pupil. All behavioral traces were lowpass \ufb01ltered to\n2.5Hz using a hamming \ufb01lter. To match the frame rate of the stimuli, all neural and behavioral traces\nwere subsequently upsampled to 30Hz using linear interpolation. Data can be downloaded from\nhttps://web.gin.g-node.org/cajal/Sinz2018_NIPS_data.\n\nStimuli The mice were presented with natural video (10s clips from both, Hollywood movies\nand rendered 3D scenes), and parametric noise clips on a standard LCD monitor. Noise movies\nconsisted of ten minutes of bandpass \ufb01ltered Gaussian noise with interleaved periods of drifting\norientation bias and ten minutes of the cosine of a low spatial frequency Gaussian process. Real\nnatural scenes included 42 min of 10s clips extracted from Hollywood action movies and the YouTube\n1M dataset [36]. Rendered natural scenes consisted of 21 min of 10s clips produced using unreal\nengine with custom scenes and programmed camera \ufb02ights. All movies were converted to grayscale\nand presented at 30Hz. Prior to feeding the data to the network, all frames were downsampled to\n36 \u00d7 64px. Videos that did not match the 16:9 ratio were center cropped.\n\nNetwork Implementation All numerical experiments and analyses were performed using Data-\nJoint [37], Numpy/Scipy [38], Matplotlib [39], Seaborn [40], Jupyter [41], PyTorch [7], and Docker\n[42]. All models were trained on NVIDIA TitanX, 1080ti, or TitanV. Code is available from\nhttps://github.com/sinzlab/Sinz2018_NIPS. The core used 3 \u00d7 3 zero-padded convolutions\n(except for the \ufb01rst layer that used 7 \u00d7 7) with 12 features in each layer. First layer \ufb01lters were regu-\nlarized with an L2 norm on the Laplace \ufb01ltered weights to encourage low frequency \ufb01lters. The \ufb01lters\nof the hidden layers were regularized using a group sparsity regularizer on all \ufb01lters corresponding\nto one output channel. Batch normalization used a momentum term of 0.1. Convolutional layers in\nthe GRU used 3 \u00d7 3 zero-padded convolutions with 36 feature channels, and no regularization. The\ninitial state of the GRU was learned as an additional parameter. Nonlinearities in the core were ELUs.\nThe readout used \ufb01ve 4 \u00d7 4 average pooling steps with stride 4, and an ELU+1 nonlinearity to keep\nthe neural responses positive. We also tried 4 \u00d7 4 with a stride of 2 but did not \ufb01nd a strong effect\non the performance. The readout weight vectors were L1 regularized. The bias of the readout was\ninitialized to match the mean response of the respective neuron. While developing the network we\nfound that this speeds up optimization, but does not affect the \ufb01nal performance. The shifter used L2\nregularization on the weight matrix. The GRU of the modulator used 50 hidden channels.\n\nTraining Schedule and Hyper-Parameter Selection Due to the large number of hyper-parameters,\nthe speci\ufb01c network and training settings were determined using a combination of grid search and\nmanual exploration on a validation set. We selected for kernel size, channels, and regularization\nconstants. We found that strong input \ufb01lter regularization helps, and sparse regularization in the\n\n4\n\n\f(b) oracle vs. model performance (c) in\ufb02uence of shifter and modulator\n(a) full model vs. linear-nonlinear\n(a) Single trial correlation of the model prediction with\nFigure 2: Performance of the model.\nthe neuronal responses across neurons for the full recurrent model and a spatio-temporal linear-\nnonlinear model equipped with a shifter and modulator. Error bars mark 95% con\ufb01dence interval. (b)\nModel test correlation against oracle correlation, which represents an upper bound of the achievable\nperformance bases solely on the stimulus. The percentage of oracle was computed from the slope of\na linear regression without offset. (c) In\ufb02uence of the single network components on the prediction\nperformance.\n\nreadout is important. When the number of channels is too large, we run into over\ufb01tting problems.\nWithin a reasonable range of kernel sizes the network is quite insensitive to the kernel size. Afterwards,\nwe used the same settings for all recordings, unless we explicitly explored the effect of different\nnetwork architectures on performance. Networks were either trained on noise movies or natural\nmovies. Each training batch contained 8 clips of 5s each, randomly selected from longer clips\nin the training set. Validation and test scores were computed on the full length clips. The pupil\nposition was standardized to mean zero and standard deviation one across the entire training set.\nThe behavioral traces were divided by their standard deviation. We used time-averaged Poisson loss\n(cid:104)\u02c6y \u2212 y log(\u02c6y + \u03b5)(cid:105)t>t0\nto train the models, with \u03b5 = 10\u221216 for numerical stability and an initial\nburn-in period of t0 = 15 frames to allow the recurrent networks to settle in from the initial state. We\noptimized the objective using ADAM [43] and a two stage training schedule with step size 0.005 and\n0.001, respectively. Each training stage \ufb01nished if either the correlation between single trials of the\nvalidation set and the model responses (using the same burn in period of 15 frames) did not improve\nover the current best result for 5 occasions checked every 4 sweeps through the training set or if the\nnumber of sweeps through the dataset exceeded 500. At the end of each training stage, the model\nwas reset to the best performing model within that stage.\n\n5 Results\n\n5.1 Performance\n\nWe trained networks on noise or natural movies from scans on three different animals (17358-5-3,\n17797-8-5, 18142-6-3). We measured the performance by the correlation between the model\nprediction and the neural responses across six 10s test clips each repeated 10 times. The networks\nreach an average correlation across neurons of 0.145 to 0.18, depending on the particular scan and\nmovie type. Note that these are single-trial predictions at 30 Hz, thus the relatively low prediction\naccuracy; we will discuss this below. We compared our recurrent network to a linear-nonlinear model\nconsisting of a 3D convolutional layer with \ufb01lter size 13 (in space and time) and 36 channels, a\nbatch norm layer, and the same readout, shifter, and modulator architecture as the recurrent model\nto allow it to account for variability unrelated to visual stimulus. This model had a comparable\nnumber of parameters as our network. Ignoring components common to both and biases the number\nof parameters were: linear-nonlinear model 133 \u00b7 36 = 79092 (3D-conv); ours 91740 total parameters\n(72 \u00b7 12 + (122 + 24\u00b7 12)\u00b7 32 in CNN, 6\u00b7 33 \u00b7 362 in GRU)). However, even though the linear-nonlinear\nnetwork had on the same order of parameter and used the same shifter or modulator components, our\nrecurrent network consistently performed better (Figure 2a).\n\n5\n\n\fCortical neurons naturally exhibit a substantial degree of variability in their responses which affects\ntheir predictability. To get an idea of the model performance relative to the best achievable perfor-\nmance, we correlated each trial in the test set with the \u201coracle\u201d estimator, computed by correlating\nthe mean over the n \u2212 1 other repeats with the remaining trial, and averaging that over all splits and\nrepeated images. We estimated the percentage-of-oracle as 100\u00d7 the slope of a linear regression\nwithout offset \ufb01tted to the oracle and model test correlations. All networks achieve 50% to 70%\nof the achievable oracle score (Figure 2b). These scores are for natural movies only, since noise\nmovies were not repeated. Note that the network performance could in principle be better than the\noracle performance, since the oracle is only computed on repeats of the stimulus and not on trial\nspeci\ufb01c-behavioral variables to which the network has access. In order to measure the contribution\nof the shifter and modulator components on the prediction performance, we trained networks with\nthose components turned off. Without shifter or modulator, the percent oracle scores were 35.7%\n(17358-5-3, 50.4% with), 56.3% (17797-8-5, 68.3% with), 44.5% (18142-6-3, 57.7% with). This\nshowed that both components improve the network performance (Figure 2c). The relative contribution\nof each component depends on the particular dataset.\n\n6 Domain transfer\n\nIn the following two sections, we explore to what extent a model trained on natural videos can predict\nneural responses and tuning properties determined by noise stimuli.\n\n6.1 Tuning\n\nWe \ufb01rst mapped receptive \ufb01elds of the networks trained on either natural movies or noise with a newly\ngenerated set of colored noise and compared them to receptive \ufb01elds of neurons mapped with reverse\ncorrelation. Figure 3a shows a selection of receptive \ufb01elds for neurons with the best prediction scores\non natural movies, along with the receptive \ufb01eld of the real neuron and the model trained on noise\nmovies. Qualitatively, the orientation, location, and general sub-\ufb01eld structure matches between the\nnetworks and neurons.\nNext, we computed direction tuning curves for the real neurons and their respective model neurons in\nmodels trained on noise and natural movies (Figure 3b). On average, both models correctly infer the\npreferred orientation, but sometimes exhibit a sign \ufb02ip in the direction. The model trained on noise\ntypically exhibits a closer match with the tuning of the real neuron. We quanti\ufb01ed this by computing\nthe distribution of the difference \u2206\u03c6 in preferred orientation between model neurons and their real\ncounterparts. We considered all neurons whose direction tuning functions had an R2 > 0.005 and\nan orientation selectivity index OSI> 0.2 (R2 > 0.002 and DSI> 0.1 for direction selectivity;\n{D,O}SI= (rp \u2212 ra)/(rp + ra) for rp, ra are the mean responses in the preferred and anti-preferred\norientation/direction), and models trained on noise and natural movies, as well as with and without\nshifter and modulator networks (Figure 3c). In all instances, the distributions are centered around zero\nwhich means that all the models predict the correct orientation on average. However, for orientation\nselectivity models on natural movies without shifter and modulator components exhibit a larger\nvariance (p < 0.03, p < 0.0015, p < 10\u221211 for the three scans using Levene\u2019 test) and slight biases\nin the median of the distribution. This indicates that accounting for confounding variables can be\nrelevant in domain transfer for neural prediction. The models trained on noise exhibit a substantially\nlower variance in \u2206\u03c6. A similar pattern is seen when quantifying the difference in preferred direction\n(Figure 3d).\n\n6.2 Limits of domain transfer\n\nWhile the network trained on natural images generally predicts the correct tuning properties of the\nneurons, there is a clear drop in the quality of tuning property prediction across stimulus domains.\nOne possible reason for this could be that a network core trained on natural movies does not provide\nthe right features to predict responses to noise. To test this hypothesis, we trained a network with\nthree readouts all referring to the same neurons and a core exclusively trained on one stimulus domain\n(Figure 4a). To ensure that the core was only trained on one stimulus domain, we stopped the gradient\nof both other readouts before the core. One readout was trained with natural movies only, one with\n\n6\n\n\f(a) Receptive \ufb01elds\n\n(b) Direction tuning curves\n\n(c) Difference in preferred orientation.\n\n(d) Difference in preferred direction.\n\nFigure 3: (a) Receptive \ufb01elds computed by reverse correlation of colored Gaussian noise with\nresponses of a network trained on neural responses to noise (top row in each group), the actual neural\nresponses (middle row), and a network trained on responses to natural video. For presentation, we\naveraged the spatio-temporal receptive \ufb01eld over the \ufb01rst 300ms. (b) Direction tuning curves of\nreal neurons (dashed) and their model counterparts trained on natural movies (pink) and (noise)\nblue. We show curves for neurons that were best predicted on natural images, among all neurons\nthat exhibited direction tuning with R2 > 0.005 and an orientation selectivity index (OSI) > 0.2.\nCurves are z-scored to match the scale. (c-d) Difference in preferred orientation (left) and direction\n(right) between neurons and model neurons trained on either natural or noise movies for model with\nshifter and modulators, and without. We only considered neurons with R2 > 0.005 and OSI> 0.2\n(R2 > 0.002 and DSI> 0.1 for direction selectivity). Lines denote the median.\n\n7\n\n\f(b) Cross-domain performance for the noise and natural movie core.\n\n(a) network con\ufb01gurations\n\n(c) Performance of readout is domain dependent.\n\nFigure 4: (a) Network con\ufb01guration for domain transfer training. Vertical squares represent readouts.\nInterrupted lines denotes gradient stopping during training. All readouts are trained on the same set\nof neurons but on different stimulus domains. (b) The core trained on natural movies performs well\non noise if the readout is trained on noise (left panel), but not vice versa (right panel). (c) Using the\nnatural movie core, performance drops when the readout is not trained on the respective domain.\nHowever, one readout trained on both domains performs as well as each readout trained dedicated on\nthe test domain. Note that the left and right bar groups in each panel refer to the same network.\n\nnoise movies only, and one with both. We also shared the grid locations of the neurons between the\nreadouts, and balanced the stimulus domains in the batches for the shared readout.\nWe \ufb01rst compared the performance of each core (trained on natural movies vs. noise) always using\nthe readout trained on the target domain (i. e. readout trained on the same stimulus domain as the test\nset). We found that both the natural movie core and the noise core perform well when testing on noise.\nIn contrast, we observe a substantial performance drop for the noise core when testing on natural\nmovies (Figure 4b). This result shows that the feature representations of the natural movie core are\nrich enough to transfer to noise stimuli, while the converse is not true. One possible confounding\nfactor for this \ufb01nding is the different stimulus presentation times between noise (20min) and natural\nmovies (\u223c1h). We ran a control experiment to show that this is not the case (see supplementary\nmaterial). We also ran a control experiment to show that there is no drop in performance between\nnetworks trained on rendered movies vs. natural movies only (see supplementary material).\nWe next turn to the setting where the readout is not trained without using stimuli from the target\ndomain. For this experiment we always use the core trained on movies, and investigate three different\nreadouts: trained on movies, trained on noise, and traind on both. We \ufb01nd that when training and\ntesting domain are not the same, there is a clear decrease in prediction performance (Figure 4c, green\n\n8\n\nNatural/Rendered Movies17358-5-3Noise Movies17358-5-317358-5-3Natural/Rendered/Noise MoviesCNN17358-5-317358-5-317358-5-3movie coreCNN17358-5-317358-5-3noise core17358-5-317358-5-317797-8-5scan0.000.050.100.150.20test correlationreadout trained and tested on noisecore trained onnoisemovies17358-5-317797-8-5scanreadout trained and tested on moviesnoisemoviestest data0.000.050.100.150.20test correlation17358-5-3readout trained onnoisebothmoviesnoisemoviestest data17797-8-5\fFigure 5: Exploration\nof the Poisson loss\nand the correlation be-\ntween prediction and\nneural\nresponse on\nthe space of readout\nweights\ninterpolated\nbetween the \u201cboth\u201d\nreadout and the ded-\nicated readouts (Fig-\nure 4a).\n\nand gray bars). This could either mean that the readout weights cannot be correctly learned from the\ndataset on one domain, or that the real neurons adapt to the particular stimulus domain and no set\nof readout weights can predict well on both domains. We investigated this question with a readout\ntrained on both stimulus domains. We found that this single readout can perform as well on both\ndomains as each dedicated readout trained on the target domain (compare blue bars to either green or\ngray). Thus, there is a network that can correctly transfer between both stimulus domains, but that\nthis network cannot be identi\ufb01ed by simply training on the natural movie dataset alone.\nTo corroborate this \ufb01nding and to determine that difference in performance is caused by the readout\nweights and not the bias terms we perform a recovery experiment where we linearly interpolate\nbetween the weights of the readout trained on both domains and the respective single domain readout\nweights (controlled by \u03bbnoise and \u03bbmovies in Figure 5). For each interpolation, we compute the\nPoisson loss and the correlation between prediction and neuronal response on the training set. The\nresulting loss surfaces are consistent with the identi\ufb01ability hypothesis and show that the network\nhas no gradient information to \ufb01nd the set of readout weights that transfers correctly between both\ndomains (origin in each panel of Figure 5).\n\n7 Summary\n\nWe presented a novel recurrent network architecture that can \ufb01t the responses of thousands of neurons\nto movies. In addition, our network also accounts for stimulus independent response variation caused\nby brain state changes and eye movements during the experiment. We demonstrated that both these\nfactors can increase the prediction performance of the network and the ability to transfer neural\nproperties between stimulus domains. To the best of our knowledge, this network is state-of-the-art\nin predicting neural responses to natural video.\nWe demonstrated that this network trained on natural movies captures neuronal tuning properties\ndetermined on noise. Finally, we showed that there is a network that transfers very well between\nboth stimulus domains, but that this network cannot be identi\ufb01ed from training on the natural movie\ndomain alone. One possible avenue to overcome this problem in the future would be to introduce the\ncorrect model biases in our architecture via a carefully chosen regularization scheme.\n\n9\n\n0.00.51.0movies0.00.51.0noiseon noise data0.00.51.0movies0.00.51.0noisePoisson losson movie data0.00.51.0movies0.00.51.0noise0.00.51.0movies0.00.51.0noisePearson correlation0.00.10.20.30.40.50.100.120.140.160.180.20\fAcknowledgments\n\nThe authors would like to thank David Klindt and Zhe Li for comments on the manuscript. Supported\nby the Intelligence Advanced Research Projects Activity (IARPA) via Department of Interior/Interior\nBusiness Center (DoI/IBC) contract number D16PC00003. The U.S. Government is authorized\nto reproduce and distribute reprints for Governmental purposes notwithstanding any copyright\nannotation thereon. Disclaimer: The views and conclusions contained herein are those of the authors\nand should not be interpreted as necessarily representing the of\ufb01cial policies or endorsements, either\nexpressed or implied, of IARPA, DoI/IBC, or the U.S. Government. Fabian Sinz is supported by the\nInstitutional Strategy of the University of T\u00fcbingen (Deutsche Forschungsgemeinschaft, ZUK 63)\nand the Carl-Zeiss-Stiftung. This work was supported in part by NSF NeuroNex grant 1707400. The\nauthors thank Vathes LLC (https://vathes.com/) for hosting the database to reproduce results of\nthis work.\n\nStatement of Financial Interest\n\nEdgar Y. Walker, Dimitri Yatsenko, Jacob Reimer, and Andreas S. Tolias hold equity ownership in\nVathes LLC which provides consulting for the framework (DataJoint) used to develop/host the data\nfor this publication.\n\nReferences\n[1] S. V. David, W. E. Vinje, and J. L. Gallant. Natural Stimulus Statistics Alter the Receptive Field\nStructure of V1 Neurons. Journal of Neuroscience, 24(31):6991\u20137006, 2004. ISSN 0270-6474.\ndoi: 10.1523/JNEUROSCI.1422-04.2004.\n\n[2] J. Fournier, C. Monier, M. Pananceau, and Y. Fr\u00e9gnac. Adaptation of the simple or complex\nnature of V1 receptive \ufb01elds to visual statistics. Nature Neuroscience, 14(8):1053\u20131060, 2011.\ndoi: 10.1038/nn.2861.\n\n[3] C. Stringer, M. Pachitariu, Ni. Steinmetz, C. Reddy, M. Carandini, and K. D. Harris. Sponta-\n\nneous behaviors drive multidimensional , brain-wide neural activity. Technical report, 2018.\n\n[4] J. Reimer, E. Froudarakis, C. R. Cadwell, D. Yatsenko, G. H. Den\ufb01eld, and A. S. Tolias. Pupil\nFluctuations Track Fast Switching of Cortical States during Quiet Wakefulness. Neuron, 84(2):\n355\u2013362, 2014. ISSN 08966273. doi: 10.1016/j.neuron.2014.09.033.\n\n[5] M. J. McGinley, M. Vinck, J. Reimer, R. Batista-Brito, E. Zagha, C. R. Cadwell, Andreas S.\nTolias, J. A. Cardin, and D. A. McCormick. Waking state: rapid variations modulate neural and\nbehavioral responses. Neuron, 87(6):1143\u20131161, 2015. ISSN 08966273. doi: 10.1016/j.neuron.\n2015.09.012.\n\n[6] Cristopher M. Niell and Michael P. Stryker. Modulation of Visual Responses by Behavioral\nState in Mouse Visual Cortex. Neuron, 65(4):472\u2013479, 2010. ISSN 08966273. doi: 10.1016/j.\nneuron.2010.01.033.\n\n[7] A. Paszke, S. Gross, S. Chintala, G. Chanan, E Yang, Z. DeVito, Z. Lin, A. Desmaison,\n\nK. Antiga, and A. Lerer. Automatic differentiation in pytorch. 2017.\n\n[8] A. S. Ecker, P. Berens, R. J. Cotton, M. Subramaniyan, G. H. Den\ufb01eld, C. R. Cadwell, S. M.\nSmirnakis, M. Bethge, and A. S. Tolias. State dependence of noise correlations in macaque\nprimary visual cortex. Neuron, 82(1):235\u201348, apr 2014. ISSN 1097-4199. doi: 10.1016/j.neuron.\n2014.02.006. URL http://www.cell.com/article/S0896627314001044/fulltext.\n\n[9] J. F A Poulet and C. C H Petersen. Internal brain state regulates membrane potential synchrony\nin barrel cortex of behaving mice. Nature, 454(7206):881\u2013885, 2008. ISSN 00280836. doi:\n10.1038/nature07150.\n\n[10] S. A. Cadena, G. H. Den\ufb01eld, E. Y. Walker, L. A. Gatys, A. S. Tolias, M. Bethge, and A. S.\nEcker. Deep convolutional models improve predictions of macaque v1 responses to natural\nimages. bioRxiv, 2017. doi: 10.1101/201764.\n\n10\n\n\f[11] William F. Kindel, Elijah D. Christensen, and Joel Zylberberg. Using deep learning to reveal\n\nthe neural code for images in primary visual cortex. arXiv:1706.06208 [cs, q-bio], 2017.\n\n[12] D. Klindt, A. S. Ecker, T. Euler, and M. Bethge. Neural system identi\ufb01cation for large\npopulations separating \u201cwhat\u201d and \u201cwhere\u201d. In Advances in Neural Information Processing\nSystems 30, Sep 2017. in press.\n\n[13] Yimeng Zhang, Tai Sing Lee, Ming Li, Fang Liu, and Shiming Tang. Convolutional neural\n\nnetwork models of V1 responses to complex patterns. bioRxiv, page 296301, 2018.\n\n[14] K Cho, B. von Merrienboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk, and Y. Ben-\ngio. Learning Phrase Representations using RNN Encoder\u2013Decoder for Statistical Machine\nTranslation. Technical report, 2014.\n\n[15] S. Ioffe and C. Szegedy. Batch Normalization : Accelerating Deep Network Training by\n\nReducing Internal Covariate Shift. Technical report, 2015.\n\n[16] Djork-Arn\u00e9 Clevert, Thomas Unterthiner, and Sepp Hochreiter. Fast and Accurate Deep\nNetwork Learning by Exponential Linear Units (ELUs). pages 1\u201314, 2015. ISSN 09226389.\ndoi: 10.3233/978-1-61499-672-9-1760.\n\n[17] G. Huang, Z. Liu, K. Q. Weinberger, and L. van der Maaten. Densely Connected Convolutional\n\nNetworks. In CVPR, 2017. doi: 10.1109/CVPR.2017.243.\n\n[18] M. Pachitariu and M. Sahani. Learning visual motion in recurrent neural networks. In F. Pereira,\nC. J. C. Burges, L. Bottou, and K. Q. Weinberger, editors, Advances in Neural Information\nProcessing Systems 25, pages 1322\u20131330. Curran Associates, Inc., 2012.\n\n[19] M. Jaderberg, K. Simonyan, A. Zisserman, and K. Kavukcuoglu. Spatial transformer networks.\nIn C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in\nNeural Information Processing Systems 28, pages 2017\u20132025. Curran Associates, Inc., 2015.\n\n[20] D. Zoccolan, B. Graham, and D. Cox. A self-calibrating, camera-based eye tracker for the\nrecording of rodent eye movements. Frontiers in Neuroscience, 4:193, 2010. ISSN 1662-453X.\ndoi: 10.3389/fnins.2010.00193.\n\n[21] J.S Stahl, A.M van Alphen, and C.I De Zeeuw. A comparison of video and magnetic search\ncoil recordings of mouse eye movements. 99(1):101\u2013110. ISSN 0165-0270. doi: 10.1016/\nS0165-0270(00)00218-1.\n\n[22] Bart van Alphen, Beerend H. J. Winkelman, and Maarten A. Frens. Three-dimensional optoki-\n\nnetic eye movements in the c57bl/6j mouse. 51(1):623\u2013630. doi: 10.1167/iovs.09-4072.\n\n[23] Y. Fu, J. M. Tucciarone, J. S. Espinosa, N. Sheng, D. P. Darcy, R. A. Nicoll, Z. J. Huang, and\nM. P. Stryker. A cortical circuit for gain control by behavioral state. Cell, 156(6):1139\u20131152,\n2014. ISSN 10974172. doi: 10.1016/j.cell.2014.01.050.\n\n[24] S. V. David and J. L. Gallant. Predicting neuronal responses during natural vision. Network:\nComputation in Neural Systems, 16(2-3):239\u2013260, 2005. ISSN 0954898X. doi: 10.1080/\n09548980500464030.\n\n[25] S. Nishimoto and J. L. Gallant. A Three-Dimensional Spatiotemporal Receptive Field Model\nExplains Responses of Area MT Neurons to Naturalistic Movies. Journal of Neuroscience, 31\n(41):14551\u201314564, 2011. ISSN 0270-6474. doi: 10.1523/JNEUROSCI.6801-10.2011.\n\n[26] R. Prenger, M. C. K. Wu, S. V. David, and J. L. Gallant. Nonlinear V1 responses to natural\nscenes revealed by neural network analysis. Neural Networks, 17(5-6):663\u2013679, 2004. ISSN\n08936080. doi: 10.1016/j.neunet.2004.03.008.\n\n[27] B. Lau, G. B. Stanley, and Y. Dan. Computational subunits of visual cortical neurons revealed by\narti\ufb01cial neural networks. Proceedings of the National Academy of Sciences, 99(13):8974\u20138979,\n2002. ISSN 0027-8424. doi: 10.1073/pnas.122173799.\n\n11\n\n\f[28] B. Vintch, J. A. Movshon, and E. P. Simoncelli. A Convolutional Subunit Model for Neuronal\nResponses in Macaque V1. The Journal of neuroscience : the of\ufb01cial journal of the Society for\nNeuroscience, 35(44):14829\u201341, 2015. ISSN 1529-2401. doi: 10.1523/JNEUROSCI.2815-13.\n2015.\n\n[29] E. Batty, J. Merel, N. Brackbill, A. Heitman, A. Sher, A. Litke, E. J. Chichilnisky, and L. Panin-\n\nski. Multilayer network models of primate retinal ganglion cells. Number Nips, 2016.\n\n[30] D. Sussillo, R. Jozefowicz, L. F. Abbott, and C. Pandarinath. LFADS - Latent Factor Analysis\n\nvia Dynamical Systems. Technical report, 2016.\n\n[31] S. R. Lehky, T. J. Sejnowski, and R. Desimone. Predicting responses of nonlinear neurons in\nmonkey striate cortex to complex patterns. The Journal of Neuroscience, 12(9):3568\u20133581,\n1992. ISSN 0270-6474.\n\n[32] J. Antol\u00edk, S. B. Hofer, J. A. Bednar, and T. D. Mrsic-\ufb02ogel. Model Constrained by Visual\nHierarchy Improves Prediction of Neural Responses to Natural Scenes. PLoS Comput Biol,\npages 1\u201322, 2016. ISSN 1553-7358. doi: 10.1371/journal.pcbi.1004927.\n\n[33] D. Zipser and R.A. A. Andersen. A back-propagation programmed network that simulates\nresponse properties of a subset of posterior parietal neurons. Nature, 331(6158):679\u2013684, 1988.\nISSN 0028-0836. doi: 10.1038/331679a0.\n\n[34] E. A. Pnevmatikakis, Y. Gao, D. Soudry, D. Pfau, C. Lace\ufb01eld, K. Pskanzer, R. Bruno, R. Yuste,\nand L. Paninski. Simultaneous Denoising, Deconvolution, and Demixing of Calcium Imaging\nData Eftychios. Neuron, 89:285\u2013299, 2016.\n\n[35] N. J. Sofroniew, D. Flickinger, J. King, and K. Svoboda. A large \ufb01eld of view two-photon\nmesoscope with subcellular resolution for in vivo imaging. eLife, 5(JUN2016):1\u201320, 2016.\nISSN 2050084X. doi: 10.7554/eLife.14472.\n\n[36] Andrej Karpathy, George Toderici, Sanketh Shetty, Thomas Leung, Rahul Sukthankar, and\nLi Fei-Fei. Large-scale video classi\ufb01cation with convolutional neural networks. In CVPR, 2014.\n\n[37] D. Yatsenko, J. Reimer, A. S. Ecker, E. Y. Walker, F. Sinz, P. Berens, A. Hoenselaar, R. J.\nCotton, A. S. Siapas, and A. S. Tolias. DataJoint: managing big scienti\ufb01c data using MATLAB\nor Python. Technical report, 2015.\n\n[38] St\u00e9fan van der Walt, S Chris Colbert, and Gael Varoquaux. The numpy array: a structure for\n\nef\ufb01cient numerical computation. Computing in Science & Engineering, 13(2):22\u201330, 2011.\n\n[39] John D Hunter. Matplotlib: A 2d graphics environment. Computing in science & engineering,\n\n9(3):90\u201395, 2007.\n\n[40] Michael Waskom, Olga Botvinnik, Drew O\u2019Kane, Paul Hobson, Saulius Lukauskas, David C\nGemperline, Tom Augspurger, Yaroslav Halchenko, John B. Cole, Jordi Warmenhoven, Julian\nde Ruiter, Cameron Pye, Stephan Hoyer, Jake Vanderplas, Santi Villalba, Gero Kunter, Eric\nQuintero, Pete Bachant, Marcel Martin, Kyle Meyer, Alistair Miles, Yoav Ram, Tal Yarkoni,\nMike Lee Williams, Constantine Evans, Clark Fitzgerald, Brian, Chris Fonnesbeck, Antony\nLee, and Adel Qalieh. mwaskom/seaborn: v0.8.1 (september 2017), September 2017.\n\n[41] Thomas Kluyver, Benjamin Ragan-Kelley, Fernando P\u00e9rez, Brian Granger, Matthias Bussonnier,\nJonathan Frederic, Kyle Kelley, Jessica Hamrick, Jason Grout, Sylvain Corlay, Paul Ivanov,\nDami\u00e1n Avila, Sa\ufb01a Abdalla, and Carol Willing. Jupyter notebooks \u2013 a publishing format for\nreproducible computational work\ufb02ows. In F. Loizides and B. Schmidt, editors, Positioning and\nPower in Academic Publishing: Players, Agents and Agendas, pages 87 \u2013 90. IOS Press, 2016.\n\n[42] Dirk Merkel. Docker: Lightweight linux containers for consistent development and deployment.\n\nLinux J., 2014(239), March 2014. ISSN 1075-3583.\n\n[43] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint\n\narXiv:1412.6980, 2014.\n\n12\n\n\f", "award": [], "sourceid": 3582, "authors": [{"given_name": "Fabian", "family_name": "Sinz", "institution": "University T\u00fcbingen"}, {"given_name": "Alexander", "family_name": "Ecker", "institution": "University of Tuebingen"}, {"given_name": "Paul", "family_name": "Fahey", "institution": "Bayl"}, {"given_name": "Edgar", "family_name": "Walker", "institution": "Baylor College of Medicine"}, {"given_name": "Erick", "family_name": "Cobos", "institution": "Baylor College of Medicine"}, {"given_name": "Emmanouil", "family_name": "Froudarakis", "institution": "Baylor College of Medicine"}, {"given_name": "Dimitri", "family_name": "Yatsenko", "institution": "Baylor College of Medicine"}, {"given_name": "Zachary", "family_name": "Pitkow", "institution": "BCM/Rice"}, {"given_name": "Jacob", "family_name": "Reimer", "institution": "Baylor College of Medicine"}, {"given_name": "Andreas", "family_name": "Tolias", "institution": "Baylor College of Medicine"}]}