{"title": "From voxels to pixels and back: Self-supervision in natural-image reconstruction from fMRI", "book": "Advances in Neural Information Processing Systems", "page_first": 6517, "page_last": 6527, "abstract": "Reconstructing observed images from fMRI brain recordings is challenging. Unfortunately, acquiring sufficient ''labeled'' pairs of {Image, fMRI} (i.e., images with their corresponding fMRI responses) to span the huge space of natural images is prohibitive for many reasons. We present a novel approach which, in addition to the scarce labeled data (training pairs), allows to train fMRI-to-image reconstruction networks also on \"unlabeled\" data (i.e., images without fMRI recording, and fMRI recording without images). The proposed model utilizes both an Encoder network (image-to-fMRI) and a Decoder network (fMRI-to-image). Concatenating these two networks back-to-back (Encoder-Decoder & Decoder-Encoder) allows augmenting the training data with both types of unlabeled data. Importantly, it allows training on the unlabeled test-fMRI data. This self-supervision adapts the reconstruction network to the new input test-data, despite its deviation from the statistics of the scarce training data.", "full_text": "From voxels to pixels and back: Self-supervision in\n\nnatural-image reconstruction from fMRI\n\nRoman Beliy\u2217\n\nGuy Gaziv\u2217\n\nDept. of Computer Science and Applied Math\n\nDept. of Computer Science and Applied Math\n\nThe Weizmann Institute of Science\n\n76100 Rehovot, Israel\n\nroman.beliy@weizmann.ac.il\n\nThe Weizmann Institute of Science\n\n76100 Rehovot, Israel\n\nguy.gaziv@weizmann.ac.il\n\nAssaf Hoogi\n\nDept. of Computer Science and Applied Math\n\nThe Weizmann Institute of Science\n\n76100 Rehovot, Israel\n\nassaf.hoogi@weizmann.ac.il\n\nFrancesca Strappini\nDept. of Neurobiology\n\nThe Weizmann Institute of Science\n\n76100 Rehovot, Israel\n\nfrancescastrappini@gmail.com\n\nTal Golan\n\nZuckerman Institute\nColumbia University\n\n10027 New York, NY, USA\ntal.golan@columbia.edu\n\nMichal Irani\n\nDept. of Computer Science and Applied Math\n\nThe Weizmann Institute of Science\n\n76100 Rehovot, Israel\n\nmichal.irani@weizmann.ac.il\n\nAbstract\n\nReconstructing observed images from fMRI brain recordings is challenging. Unfor-\ntunately, acquiring suf\ufb01cient \u201clabeled\u201d pairs of {Image, fMRI} (i.e., images with\ntheir corresponding fMRI responses) to span the huge space of natural images is\nprohibitive for many reasons. We present a novel approach which, in addition to\nthe scarce labeled data (training pairs), allows to train fMRI-to-image reconstruc-\ntion networks also on \u201cunlabeled\u201d data (i.e., images without fMRI recording, and\nfMRI recording without images). The proposed model utilizes both an Encoder\nnetwork (image-to-fMRI) and a Decoder network (fMRI-to-image). Concatenating\nthese two networks back-to-back (Encoder-Decoder & Decoder-Encoder) allows\naugmenting the training with both types of unlabeled data. Importantly, it allows\ntraining on the unlabeled test-fMRI data. This self-supervision adapts the re-\nconstruction network to the new input test-data, despite its deviation from the\nstatistics of the scarce training data.\n\nProject Website: http://www.wisdom.weizmann.ac.il/~vision/ssfmri2im/\n\nIntroduction\n\n1\nDeveloping a method for high-quality reconstruction of seen images from the corresponding brain\nactivity is an important milestone towards decoding the contents of dreams and mental imagery\n(Fig 1a). In this task, one attempts to solve for the mapping between fMRI recordings and their\ncorresponding natural images, using many \u201clabeled\u201d {Image, fMRI} pairs (i.e., images and their\ncorresponding fMRI responses). A good fMRI-to-image decoder is one that will generalize well to\nreconstruction of new never-before-seen images from new fMRI recordings (we refer to these as\n\n\u2217Equal contribution\n\n33rd Conference on Neural Information Processing Systems (NeurIPS 2019), Vancouver, Canada.\n\n\fFigure 1: Our proposed method. (a) The task: reconstructing images from evoked brain activity, inferred\nfrom fMRI responses. (b), (c) Supervised training for decoding (b) and encoding (c) using limited training\npairs. This gives rise to poor generalization. (d), (e) Illustration of our added self-supervision, which enables\ntraining on \u201cunlabeled images\u201d (any natural image with no fMRI recording \u2013 (d)), and on the \u201cunlabeled fMRI\u201d\n(fMRI data without any corresponding images \u2013 (e)). In particular, the latter allows adapting the decoder to the\nstatistics of the target test-fMRI despite not having any information about their corresponding images.\n\n\u201ctest-data\u201d or \u201ctest-fMRI\u201d). However, the lack in \u201clabeled\u201d training data limits the generalization\npower of today\u2019s fMRI decoders. Acquiring a large number of labeled pairs {Image, fMRI} is\nprohibitive, due to the limited time a human can spend in an MRI scanner. As a result, most datasets\nare limited to a few thousands of such pairs. Such limited samples cannot span the huge space of\nnatural images, nor the space of their fMRI recordings. Moreover, the poor spatio-temporal resolution\nof fMRI signals, as well as their low Signal-to-Noise Ratio (SNR), reduce the reliability of the already\nscarce labeled training data. Lastly, the train-set and test-set of the fMRI data often differ in their\nstatistical properties, speci\ufb01cally in their SNR. This SNR discrepancy is due to averaging a different\nnumber of repeated recordings per image (typical of many fMRI datasets). It therefore introduces an\nadditional challenge of \u2018domain transfer/adaptation\u2019, which makes generalization even harder, and\naffects the performance of current decoding methods.\nPrior work in image reconstruction from fMRI. The task of reconstructing a visual stimulus from\nfMRI has been approached by a number of methods which can broadly be classi\ufb01ed into three\nfamilies: (i) Linear regression between fMRI data and handcrafted image-features (e.g., Gabor\nwavelets) [1, 2, 3], (ii) Linear regression between fMRI data and Deep (CNN-based) image-features\n(e.g., using pretrained AlexNet) [4, 5, 6], and (iii) End-to-end Deep Learning [7, 8, 9, 10].\nThe \ufb01rst two regression-based methods compute a linear model that relates fMRI voxels to image\nfeature representation. This can be done by either linearly predicting each voxel\u2019s responses from\nthe image features [1, 2, 3, 5], or by linearly mapping voxel responses to image features from\nwhich the image can be easily recovered [4, 6]. The feature representation is chosen such that it\nclosely mimics the neural activity in the visual cortex, with the hope that a simple model (like\nlinear regression) will suf\ufb01ce to capture the remaining mapping. Methods in the second category\nbene\ufb01ted from utilizing data-driven learned features from leading CNN models trained for natural\nimage classi\ufb01cation [5, 11, 12, 13, 7, 14, 15]. The last category refers to recent attempts to train\nhigh-complexity deep models which directly decode an fMRI recording into its corresponding image\nstimulus. To our best knowledge, methods [6] and [8] are the current state-of-the-art in this \ufb01eld.\n\n2\n\nReconstructed imagesImages????fMRIFindfMRI-to-imageDecoderUnsupervised training with unlabeled dataLossAdapt to statistics of Test DatatestfMRI testfMRI ANY image (cid:28)(cid:24)(cid:28)(cid:24)Supervised training(on limited paired data)(cid:28)(cid:28)Poor generalization(cid:28)(cid:24)Adapt to statistics of Natural Images(a)(b)(c)(d)(e)DecodingEncodingfMRI-to-imageDecoder\fFigure 2: Adding un-\nsupervised\ntraining\non unlabeled data im-\nproves reconstruction.\n(Left column): the images\npresented to the human.\n(Middle column): Recon-\nstruction using the training\npairs only (Fig 1b).\n(Right column):\nRecon-\nstruction when\nadding\nunsupervised training on\nunlabeled data (Fig 1d,e).\nExample results are shown\nfor\ntwo fMRI datasets:\n\u2018fMRI on ImageNet\u2019 [16]\nand \u2018vim-1\u2019 [1].\n\nAll these methods are supervised (i.e., train their decoder on pairs of {Image, fMRI}), hence suffer\nfrom the very limited training data. Purely supervised models are prone to poor generalization to new\ntest-data (fMRI of new images). To overcome this problem, recent methods [6, 8, 9, 10] constrain the\nreconstructed image to have natural-image statistics by introducing Generative Adversarial Networks\n(GANs) into their decoder. These methods gave leap advancement in reconstruction quality from\nfMRI, and tend to produce natural-looking images. Nevertheless, despite their pleasant natural\nappearance, their reconstructed images are often unfaithful to the actual images underlying the\ntest-fMRI (see Fig 5a).\nWe present a new approach to overcome the inherent lack of training data and the discrepancy\nbetween the train/test statistics, by introducing self-supervision using unlabeled data. Our approach\nis illustrated in Fig 1. We train two types of networks: an Encoder E, to map natural images to\ntheir corresponding fMRI response, and a Decoder D, to map fMRI recordings to their correspond-\ning images. Concatenating those two networks back-to-back, E-D, yields a combined network\nwhose input and output are the same image (Fig. 1d). This allows for unsupervised training on\nunlabeled images (i.e., images without fMRI recordings, e.g., 50,000 randomly sampled natural\nimages in our experiments). Such self-supervision adapts the network to the statistics of never-\nbefore-seen images. Moreover, concatenating our two networks the other way around, D-E, yields a\ncombined network with the same shared weights as E-D, but whose input and output are now an fMRI\nsignal (Fig. 1e). This allows unsupervised training on unlabeled fMRI. Speci\ufb01cally, those unla-\nbeled fMRI samples can be legitimately drawn from the test-fMRI cohort, while their corresponding\nimages (\u201ctest-images\u201d) are excluded from training (Fig 1e).\nTraining on these unlabeled test-fMRI (without their images) is a key feature of our method:\nit enables to adapt the network to the statistics of the new (unlabeled) test-data. Learning the statistics\nof the test-fMRI directly addresses the lack in labeled training data and the train/test statistics\ndiscrepancy. Note that our \u201ctraining on test data\u201d, which may seem \u201dillegal\u201d at \ufb01rst sight, is in fact\nvalid. It refers only to training on unlabeled samples from the Decoder\u2019s input space (test-fMRI),\nwhereas the test-images (the \u201clabels\u201d) are never used at any stage of the training.\nFig. 2 exempli\ufb01es the power of adding unsupervised training on unlabeled data. Notably, we found\nthe unsupervised training on the unlabeled test-fMRI (Fig 1e) to provide the greatest boost in\nperformance.\nUnsupervised training on unlabeled natural images was also recently proposed in [8], where they used\nthese images to produce additional surrogate fMRI-data to train their model. This, however, does not\nhelp to adapt the network to the statistics of the new test-fMRI. To the best of our knowledge, we\nare the \ufb01rst to provide an approach for adding self-supervised training on unlabeled fMRI. This self-\n\n3\n\nGround truthSupervisedonly (Fig. 1b)Adding self-supervision(Fig. 1d,e)Ground truthSupervisedonly (Fig. 1b)\u2018fMRI on ImageNet\u2019\u2018vim-1\u2019Adding self-supervision(Fig. 1d,e)\fsupervision provides an improvement in decoding of never-before-seen images from brain activity,\ndespite the very limited training data.\nInsuf\ufb01cient training data and domain-adaptation [17] are the focus of many recent machine learning\nworks, including transfer learning [18], unsupervised and self-supervised learning [19, 20, 21, 22],\nsemi-supervised and transductive learning [23, 24, 25, 26]. Nevertheless, these approaches often\nassume that there is suf\ufb01cient labeled data in the \u2018source domain\u2019, and that the problem is the\ngeneralization to the unlabeled \u2018target domain\u2019. This, however, is not the case here. There is too little\ndata (both labeled or unlabeled) to work with in this challenge. The approach we took is inspired by\nthe recent advances in \u201cDeep-Internal-Learning\u201d [27, 28, 29], which train an image-speci\ufb01c network,\nat test time, on the test-image alone, without requiring any prior training examples. Our approach\ncombines ideas from Internal-Learning with supervised-learning, to get the best of both worlds.\nOur contributions are therefore several-fold:\n\u2022 We propose a new approach to handle the inherent lack in Image-fMRI training data.\n\u2022 To the best of our knowledge, we are the \ufb01rst to suggest an approach for self-supervised training\n\u2022 We demonstrate the power and versatility of our approach by applying it (with the same architecture\nand same hyperparameters) to two very different fMRI datasets. We achieve competitive results in\nimage reconstruction on both datasets (\u2018fMRI on ImageNet\u2019 [16], and \u2018vim-1\u2019 [1]). Most methods,\nincluding those we compared our results with, are adapted to only one dataset.\n\non unlabeled fMRI data (with no images), and in particular, on the test-fMRI data.\n\n2 Method overview\nOur training consists of two phases which are illustrated in Fig 3. In the \ufb01rst phase, we apply\nsupervised training of the Encoder E alone. We train it to predict the fMRI responses of input images\nusing the image-fMRI training pairs (Fig 3a). In the second phase, we use the pretrained Encoder\n(from the \ufb01rst phase) and train the Decoder D, keeping the weights of E \ufb01xed. D is trained jointly\nusing both the labeled and the unlabeled data, simultaneously. Each training batch consists of three\ntypes of training data: (i) labeled image-fMRI pairs from the training set (Fig 1b), (ii) unlabeled\nnatural images (Fig 1d), and (iii) unlabeled fMRI (Fig 1e).\nSpeci\ufb01cally, we draw the unlabeled images from a large external database of 50K ImageNet images,\nwhich is disjoint to the considered image-fMRI dataset. This promotes adaptation of the Decoder\nto the statistics of natural images. The unlabeled fMRI data is drawn from the unlabeled test-fMRI\ncohort (without any test-images, i.e., without their \u201clabels\u201d). This promotes adaptation of the Decoder\nto the statistics of the fMRI test data. Once completed, inference of test stimuli is carried out by\nfeeding-forward the test-fMRI through the trained Decoder.\nNote that our \u201ctraining on test data\u201d, which may seem \u201dillegal\u201d at \ufb01rst sight, is in fact valid. It\nrefers only to training on unlabeled samples from the Decoder\u2019s input space (test-fMRI), whereas\nthe test-images (the \u201clabels\u201d) are never used at any stage of the training.\nThe motivation for using two training phases is to allow the Encoder to converge at the \ufb01rst phase,\nand then serve as strong guidance for the more severely ill-posed decoding task, which is the focus of\nthe second phase. The weights of the Encoder are kept \ufb01xed during the Decoder training, to ensure\nthat the Encoder\u2019s output representation does not diverge from predicting fMRI responses by the\nunsupervised training objectives 1d,e.\nWe next describe each phase in more detail. We start by supervised training of the Encoder.\n2.1 The Encoder E (Images \u2192 fMRI)\nThe training of the Encoder is illustrated in Fig. 3a. Let \u02c6r = E (s) denote the encoded fMRI response\nfrom image, s, by Encoder E. We de\ufb01ne fMRI loss by a convex combination of mean square error\nand cosine proximity with respect to the ground truth fMRI, r. The fMRI loss is de\ufb01ned as:\n\nLr (\u02c6r, r) = \u03b1(cid:107)\u02c6r \u2212 r(cid:107)2 \u2212 (1 \u2212 \u03b1) cos (\u2220 (\u02c6r, r)) ,\n\n(1)\n\nwhere \u03b1 is a hyperparameter set empirically (see Implementation Details). We use this loss for\ntraining the Encoder E. However, this loss is also used to de\ufb01ne the Decoder-Encoder loss (unlabeled\nfMRI) on which we detail later.\n\n4\n\n\fFigure 3: Training phases & Architecture.\n(a) The \ufb01rst training phase: Supervised training of the\nEncoder with {Image, fMRI} pairs. (b) Second phase: Training the Decoder simultaneously with 3 types of\ndata: {Image, fMRI} pairs (supervised examples), unlabeled natural images (self-supervision), and unlabeled\ntest-fMRI (self-supevision). Note that the test-images are never used for training. The pretrained Encoder from\nthe \ufb01rst training phase is kept \ufb01xed in the second phase. (c) Encoder and Decoder architectures. BN, US, and\nReLU stand for batch normalization, up-sampling, and recti\ufb01ed linear unit, respectively.\n\nNotably, in the considered fMRI datasets, the subjects who participated in the experiments were\ninstructed to \ufb01xate at the center of the images. Nevertheless, eye movements were not recorded\nduring the scans thus the \ufb01xation performance is not known. To accommodate the center-\ufb01xation\nuncertainty, we introduced random shifts of the input images during Encoder training. This resulted in\na substantial improvement in the Encoder performance and subsequently in the image reconstruction\nquality. Upon completion of Encoder training, we transition to training the Decoder together with the\n\ufb01xed Encoder.\n2.2 The Decoder D (fMRI \u2192 Images)\nThe training loss of our Decoder consists of three main losses illustrated in Fig. 3b:\n\nLD + LED + LDE.\n\n(2)\nLD is a supervised loss on training pairs of image-fMRI. LED and LDE are unsupervised losses\non unlabeled images (without fMRIs) and unlabeled fMRIs (without images). All 3 components\nof the loss are normalized to have the same order of magnitude (all in the range [0, 1], with equal\nweights), to guarantee that the total loss is not dominated by any individual component. We found our\nreconstruction results to be relatively insensitive to the exact balancing between the three components\n(see Supplementary-Material).\nWe next detail each component of the loss.\nDecoder Supervised Training is illustrated in Fig. 1b. Given training pairs {(r, s)}, the supervised\nloss LD is applied on the decoded stimulus, \u02c6s = D (r), and is de\ufb01ned via the image reconstruction\nobjective, Ls, as\nLs consists of losses on image RGB values, LRGB, and its features, Lf eatures. We denote the\nfeatures extracted from an image, s, by \u03d5 (s), and chose \u03d5 to be pretrained a feature-extractor.\nSpeci\ufb01cally we used activations from the \ufb01rst and the second convolutional layers of VGG19 [30].\nThe Image loss for a reconstructed image \u02c6s reads:\n\nLD = Ls (\u02c6s, s) .\n\nLs (\u02c6s, s) = LRGB (\u02c6s, s) + Lf eatures (\u02c6s, s) + R (\u02c6s)\n\nLRGB (\u02c6s, s) \u221d (cid:107)\u02c6s \u2212 s(cid:107)1 , Lf eatures (\u02c6s, s) \u221d (cid:107)\u03d5 (\u02c6s) \u2212 \u03d5 (s)(cid:107)2 , R (\u02c6s) \u221d T V (\u02c6s) .\n\nThe last term corresponds to total variation (TV) regularization of the reconstructed image, \u02c6s = D (r).\nIn addition to de\ufb01ning the Decoder supervised loss, the Image loss is also used to de\ufb01ne the Encoder-\nDecoder loss (unlabeled images) explained later.\n\n5\n\n(3)\n\n(4)\n\n(cid:24)(cid:28)(cid:28)(cid:28)(cid:28)(cid:24)(cid:28)(cid:24)(cid:24)(cid:28)(cid:28)(cid:28)(cid:28)(cid:24)(cid:28)(cid:24)(cid:24)(cid:28)(cid:28)(cid:28)(cid:28)(cid:24)(cid:28)(cid:24)PretrainedAlexNetconv1stride 2conv3x3stride 2ReLUBNconv3x3stride 2ReLUBNFCconv3x3ReLU2x USBNconv3x3ReLU2x USBNFCconv3x3ReLU2x USBNEncoderDecoderconv3x3Sigmoid(cid:24)(cid:28)(cid:28)(cid:28)(cid:28)(cid:24)(cid:28)(cid:24)(cid:24)(cid:28)(cid:28)(cid:28)(cid:28)(cid:24)(cid:28)(cid:24)(cid:24)(cid:28)(cid:28)(cid:28)(cid:28)(cid:24)(cid:28)(cid:24)(a)EncoderDecoder(c)Encoder training (phase I)Architecture(b)Decoder training (phase II)(Supervised)\ud835\udca7\ud835\udc6b(Unsupervised)\ud835\udca7\ud835\udc6c\ud835\udc6b(Unsupervised)\ud835\udca7\ud835\udc6b\ud835\udc6c\fWe now detail on the crux of our method: Unsupervised training with unlabeled data.\nEncoder-Decoder training on unlabeled Natural Images is illustrated in Fig. 1d. This objective\nenables to train on any desired unlabeled image, well beyond the 1200 images included in the\nmain fMRI dataset. To train on images without corresponding fMRI responses, we map images to\nthemselves through Encoder-Decoder transformation,\n\ns (cid:55)\u2192 \u02c6sED = D (E (s)) .\n\nThe unsupervised component LED of the loss in Eq 2 on unlabeled images, s, reads:\n\nLED = Ls (\u02c6sED, s) ,\n\nwhere Ls is Image loss de\ufb01ned in Eq 3.\nDecoder-Encoder training on unlabeled test fMRI is illustrated in Fig. 1e. Adding this objective\ngreatly improved our reconstruction quality compared to training on paired samples only. To train\non fMRI data without corresponding images, we map an fMRI response to itself through Decoder-\nEncoder transformation:\n\nr (cid:55)\u2192 \u02c6rDE = E (D (r)) .\n\nThis yields the following unsupervised component LDE of the loss in Eq 2 on unlabeled fMRI\nresponses r:\nwhere Lr is fMRI loss de\ufb01ned in Eq 1.\nImportantly, the fMRI samples which we used here were drawn from the test cohort. This enables\nto adapt the Decoder to the statistics of the test-fMRI data (which we want to decode). The same\ntest-fMRI data is subsequently used at inference.\n\nLDE = Lr (\u02c6rDE, r) ,\n\n2.3\n\nImplementation details\n\nWe focused on 112x112 RGB or grayscale image reconstruction (depending on the dataset), although\nour method works well also on other resolutions.\nArchitectures of the Encoder and the Decoder are illustrated in Fig. 3c. For the Decoder we used\na fully connected layer to transform and reshape the vector-form fMRI input into 64 feature maps\nwith spatial resolution 14x14. This representation is then followed by three blocks, each consists of:\n(i) 3x3 convolution with unity stride, 64 channels, and ReLU activation, (ii) x2 up-sampling, and\n(iii) batch normalization. To yield the output image we \ufb01nally performed an additional convolution,\nsimilar to the preceding ones, but with three channels to represent colors, and a sigmoid activation\nto keep the output values in the 0-1 range. We used Glorot-normal[31] to initialize the weights.\nThe design of the Encoder consists of feature extraction using pretrained AlexNet conv1 weights,\nfollowed by batch normalization. The next operations include three blocks of 3x3 convolution with\n32 channels, ReLU activation with stride 2, and batch normalization. Lastly, we use a fully connected\nlayer to bring the output to voxel space. We initialized the weights using Glorot normal initializer.\nHyperparameter tuning. We trained the Encoder with \u03b1 = 0.9 using SGD optimizer for 80 epochs\nwith an initial learning rate of 0.1, with a prede\ufb01ned learning rate scheduler. During Decoder\ntraining with supervised and unsupervised objectives, each training batch contained 60% paired data\n(supervised training), 20% unlabeled natural images (without fMRI), and 20% unlabeled test-fMRI\n(without images). We trained the Decoder for 150 epochs using Adam optimizer with an initial\nlearning rate of 1e-3, and 80% learning rate drop after every 30 epochs.\nRuntime. Our system completes the two-stage training within approximately 15 min using a single\nTesla V100 GPU while inference (decoding) is performed in real time.\nExperimental datasets. We experimented with two publicly available (and very different) benchmark\nfMRI datasets, using the same architectures and hyperparameters: (i) fMRI on ImageNet [16], and\n(ii) vim-1 [1]. These datasets provide fMRI recordings paired with their corresponding underlying\nimages. Subjects were instructed to \ufb01xate at a cross located at center of the presented images. \u2018fMRI\non ImageNet\u2019 comprises 1250 distinct ImageNet images drawn from 200 selected categories. The\ntrain- and test-fMRI data consist of 1 and 35 (repeated recordings) per presented stimulus, respectively.\nFifty image categories provided the \ufb01fty test images, one from each category. The remaining 1200\n\n6\n\n\fwere de\ufb01ned as train set (with only one fMRI recording). We considered approximately 4500 voxels\nfrom the visual cortex provided by the authors of [16]. \u2018vim-1\u2019 comprises 1870 distinct grayscale\nimages. fMRI was recorded (i) twice for 1750 images and de\ufb01ned the training data, and (ii) 13 times\nfor the remaining 120 images, de\ufb01ning the test data.\nWe screened approximately 8500 out of the 50K recorded voxels by their SNR. We used additional\n50K unlabeled natural images from ImageNet [32] validation data for the unsupervised training on\nunlabeled images (Encoder-Decoder objective, Fig. 1d). We veri\ufb01ed that the images in our additional\nunlabeled external dataset, are distinct from those in the \u201cfMRI on ImageNet\u201d.\nPerformance evaluation. The reconstruction quality of images from fMRI was assessed both\nvisually and objectively, and was compared with the two leading methods [6, 8] (Fig. 5). The\nsimilarity measure was based on correlating pixel values between the reconstructed image and an\noriginal image. However, the absolute correlation value on its own is meaningless, and cannot be\ncompared across different types of reconstructions, because of its sensitivity to variations in edge\nintensity, edge misalignments, etc. For example, when the edges of the reconstructed image are not\naligned with those of the ground-truth image (as in the reconstructed white goat in Fig. 4d), standard\nimage-to-image similarity measures will favor a blurrier version of the reconstructed image (e.g.,\nthe goat in Fig. 4c) over a sharp one (the goat in Fig. 4d). To alleviate this inherent bias, we used\nan objective image-reconstruction quality measure by computing its \u2018correct-identi\ufb01cation rate\u2019 in\na multi-image identi\ufb01cation task (as proposed in [6]). The correlation measure, while not ideal,\nwould still produce higher correlation value with the ground-truth image, than with other random\nsharp images. For each reconstructed image, the task is to identify its ground truth image among\nn candidate images (n= 2, 5 or 10), one being the true ground truth, while the rest were randomly\nselected. This identi\ufb01cation was based on the same correlation measure (between the reconstructed\nimage and each candidate image). The candidate image which scored the max Pearson correlation\nwas determined to be the identi\ufb01ed \u2018ground truth\u2019.\nBecause of the randomness in our training process we repeated the analysis multiple times and\naveraged over the reconstructed images: 20 runs for \u2018fMRI on ImageNet\u2019, and 10 runs for \u2018vim-1\u2019.\n\n3 Experimental results\nFig. 2 shows our results with the proposed method, which includes the combined supervised and\nunsupervised training. These results (in red frames) are contrasted with the results obtainable when\nusing supervised training only (with the 1200 labeled training pairs). All the displayed images were\nreconstructed from the test-fMRI. The red-framed images show many faithfully-reconstructed shapes,\ntextures, and colors, which depict recognizable scenes and objects. In contrast, using the supervised\nobjective alone led to reconstructions that were considerably less faithful and recognizable (middle\ncolumns in Fig. 2). The reconstructions of the entire test cohort (50 images) can be found in the\nSupplementary-Material.\nAblation study of the method components\nFig. 4 shows an analysis of the merit of our unsupervised training. Our complete method, which\nincludes training on unlabeled images and on unlabeled test-fMRI is compared against three baselines:\n(i) A purely supervised approach (Fig. 1b), relying only on image-fMRI pairs (Fig. 4b). (ii) Adding\nalso unsupervised training on many external unlabeled images (Fig. 4c). Our results suggest a\ndiscernible albeit moderate improvement due to this objective. (iii) Adding also unsupervised training\non unlabeled fMRI data from the test-fMRI cohort (Fig. 4d). This provides a dramatic improvement\nin the results, However, excluding the single unlabeled target-fMRI of the speci\ufb01c test-image\n(Fig. 4e) leads to a marked degradation in reconstruction quality. This indicates the importance of\nadapting our model to the actual test-fMRI which is designated for inference.\nWe evaluated the reconstruction quality of each component of our method using n-way identi\ufb01cation-\ntask based on pixel-level similarity of the reconstructed images and candidate ground truth images.\nThis evaluation con\ufb01rmed the qualitative trend of better reconstruction by the full method compared\nto the ablated versions 1b,d,e, at varying number of candidate ground-truth images: n = 2, 5, 10 (see\nPerformance Evaluation). For a 2-way identi\ufb01cation task (detecting the source of a reconstructed\nimage among two candidate images) we report average scores of 80.1% for supervised training-only,\n83.2% when adding the training on additional unlabeled images, and 85.3% for the full method. The\nidenti\ufb01cation accuracy dropped to 84.1% when the target test-fMRI was excluded from training.\n\n7\n\n\fFigure 4: Ablation of method compo-\nnents (visual & quantitative).\n(b) Supervised training on {Image, fMRI}\n(c) Adding unsupervised\npairs (Fig.1b).\ntraining, but only on unlabeled images\n(d) Adding also unsupervised\n(Fig.1d).\ntraining on the unlabeled test-fMRI (Fig.1e).\n(e) Same as (d), but now the single unla-\nbeled test-fMRI underlying the test-image\n(i.e., the \u201ctarget-fMRI\u201d) is omitted from the\nself-supervision on the test-fMRI cohort.\n\nA major factor underlying the performance leap when training on unlabeled test-fMRI (incorporating\nLDE) was the repeat-count of test-fMRI recordings. Averaging over multiple repeats results in fMRI\nsamples with higher SNR, compared to those in the training data which only have a single repeat. Our\nablation studies showed that reconstruction improves as the number of averaged test-fMRI repeats\nincreases (see Supplementary-Material). This shows that our D-E architecture exploits the better SNR\nof the test-fMRI and adapts D to the statistics of the test-fMRI, which deviates from the statistics\n(SNR) of the training data.\nComparison with state-of-the-art methods\nWe compared our results both visually and quantitatively against the two leading methods: Shen et\nal. [6]) and St-Yves et al. [8] \u2013 each on its relevant dataset.\nVisual comparison. Fig 5a,b compares the results of our method with the corresponding ones\nproposed in [6, 8]. Each of these methods focused on one speci\ufb01c fMRI dataset, either \u2018fMRI on\nImageNet\u2019 [16] or \u2018vim-1\u2019 [1]. Both methods used GANs as natural image priors to increase their\ngeneralization power when having very limited training data, resulting in natural-looking images\nin some cases however substantially deviant from the actual images underlying the fMRI (Shen et\nal. [6]) and/or low quality (St-Yves et al. [8]). Our method seems to better reconstruct shapes, details\nand global layout in the reconstructed images than [6, 8]. This is supported visually and numerically.\nQuantitative comparison. We report quantitative objective comparisons of the reconstructed images\nby our method and those by [6, 8] in Fig. 5c,d. These panels show the correct-identi\ufb01cation rate\n(within a method) for n-way classi\ufb01cation tasks for n = 2, 5, 10 (see Performance Evaluation). We\nevaluate our method and two variants of the method of [6] on the \u2018fMRI on ImageNet\u2019 benchmark\ndataset (Fig. 5c). Our method scored 85.3% mean identi\ufb01cation accuracy, competing favorably\nagainst both variants of [6] by a margin of at least 5% across all task dif\ufb01culty levels (n = 2, 5, 10).\nWe repeated the analysis for \u2018vim-1\u2019 fMRI dataset (Fig. 5d), where our method scored accuracy of\n70.5% (for n = 2), outperforming the method from [8] by at least 3% across dif\ufb01culty levels. Taken\ntogether, our method competes favorably and slightly outperform state-of-the-art methods. This\nadvantage holds at least with respect to the two considered datasets, and is robust to varying dif\ufb01culty\nlevels of the identi\ufb01cation task.\n\nConclusion\nThis work highlights the importance of self-supervised training on unlabeled input test data. This\naddresses the inherent lack in labeled (supervised) training data, and the discrepancy between the\n\n8\n\n2-wayIdentification Accuracy (cid:31)(cid:30)(cid:29)01020304050607080905-way10-wayChance levelGround truthAdding unsupervised training on unlabeled data(a)(b)(b)Supervisedtraining only (Fig. 1b)(e)Full method without thetarget-fMRI (c)(c)Full method without unlabeled fMRI(d)(d)(e)Full method\fFigure 5: Comparison with state-of-the-art methods. (a), (b) Visual comparison with [6, 8] \u2013 each\ncompared on its relevant dataset. Our method reconstructs shapes, details and global layout in images better\nthan the leading methods. (c), (d) Quantitative comparisons of identi\ufb01cation accuracy (per method) in an n-way\nidenti\ufb01cation task (see text for details). 95% Con\ufb01dence Intervals shown on charts.\n\nstatistics of the train/test data. Our experiments show that self-supervised training on unlabeled\ntest-fMRI (without using any test-images for training) has a dramatic effect on the decoded images.\nIt further has a stronger effect than self-supervised training on unlabeled natural images only. Particu-\nlarly, including self-supervision on the target-fMRI shows the highest impact on the reconstruction\nof the corresponding target image. These suggest the importance of adapting the network to the\nstatistics of the input test data.\nWhile image reconstruction can eventually become a strong neuroscienti\ufb01c tool, this is not the\nfocus of the current work. This work highlights a new learning method, which is exempli\ufb01ed on a\ndif\ufb01cult neuroscienti\ufb01c problem, but is not limited to it. The characteristics of the fMRI-inference\nproblem are common to other ill-posed learning tasks where labeled training data is scarce, while\nhigh generalization power is desired. Adapting to the statistics of the target test data may be useful\nfor promoting generalization for those other problem areas as well.\n\nAcknowledgments\n\nThis project has received funding from the European Research Council (ERC) under the European\nUnion\u2019s Horizon 2020 research and innovation programme (grant agreement No 788535).\n\nAuthor Contributions\n\nR.B. and G.G. designed the experiments. R.B. implemented the network and conducted the image-\nreconstruction experiments. G.G. designed and wrote the paper, and analyzed the fMRI data. A.H.\nconducted reconstruction quality analyses. F.S. and T.G. provided guidance on fMRI preprocessing.\nM.I. conceived the original idea and supervised the project. All authors discussed the results and\ncommented on the manuscript.\n\n9\n\n(a)(b) (c) (d) \u2018fMRI on ImageNet\u2019\u2018vim-1\u2019Ground truthGround truthOursSt-Yves et al.Shen et al.OursShen et al.Shen et al. Image PriorOursChance levelSt-Yves et al.Chance levelOursIdentification Accuracy (cid:31)(cid:30)(cid:29)Identification Accuracy (cid:31)(cid:30)(cid:29)0102030405060708001020304050607080902-way5-way10-way2-way5-way10-way\fReferences\n[1] K. N. Kay, T. Naselaris, R. J. Prenger, and J. L. Gallant, \u201cIdentifying natural images from human brain\n\nactivity,\u201d Nature, vol. 452, pp. 352\u2013355, 3 2008.\n\n[2] T. Naselaris, R. J. Prenger, K. N. Kay, M. Oliver, and J. L. Gallant, \u201cBayesian Reconstruction of Natural\n\nImages from Human Brain Activity,\u201d Neuron, vol. 63, pp. 902\u2013915, 9 2009.\n\n[3] S. Nishimoto, A. T. Vu, T. Naselaris, Y. Benjamini, B. Yu, and J. L. Gallant, \u201cReconstructing visual\nexperiences from brain activity evoked by natural movies.,\u201d Current biology : CB, vol. 21, pp. 1641\u20136, 10\n2011.\n\n[4] H. Wen, J. Shi, Y. Zhang, K.-H. Lu, J. Cao, and Z. Liu, \u201cNeural Encoding and Decoding with Deep\n\nLearning for Dynamic Natural Vision,\u201d Cerebral Cortex, vol. 28, pp. 4136\u20134160, 12 2018.\n\n[5] U. Guclu, M. A. J. van Gerven, U. G\u00fc\u00e7l\u00fc, and M. A. J. van Gerven, \u201cDeep Neural Networks Reveal a\nGradient in the Complexity of Neural Representations across the Ventral Stream,\u201d Journal of Neuroscience,\nvol. 35, pp. 10005\u201310014, 7 2015.\n\n[6] G. Shen, T. Horikawa, K. Majima, and Y. Kamitani, \u201cDeep image reconstruction from human brain activity,\u201d\n\nPLOS Computational Biology, vol. 15, p. e1006633, 1 2019.\n\n[7] G. Shen, K. Dwivedi, K. Majima, T. Horikawa, and Y. Kamitani, \u201cEnd-to-end deep image reconstruction\n\nfrom human brain activity,\u201d bioRxiv, p. 272518, 2018.\n\n[8] G. St-Yves and T. Naselaris, \u201cGenerative Adversarial Networks Conditioned on Brain Activity Reconstruct\n\nSeen Images,\u201d bioRxiv, p. 304774, 2018.\n\n[9] K. Seeliger, U. G\u00fc\u00e7l\u00fc, L. Ambrogioni, Y. G\u00fc\u00e7l\u00fct\u00fcrk, and M. A. van Gerven, \u201cGenerative adversarial\nnetworks for reconstructing natural images from brain activity,\u201d NeuroImage, vol. 181, pp. 775\u2013785, 2018.\n[10] Y. Lin, J. Li, H. Wang, and S. Jiao, \u201cDCNN-GAN: Reconstructing Realistic Image from fMRI,\u201d tech. rep.,\n\n2019.\n\n[11] T. Horikawa and Y. Kamitani, \u201cHierarchical Neural Representation of Dreamed Objects Revealed by Brain\nDecoding with Deep Neural Network Features,\u201d Frontiers in Computational Neuroscience, vol. 11, 2017.\n[12] H. Wen, J. Shi, W. Chen, and Z. Liu, \u201cDeep Residual Network Predicts Cortical Representation and\n\nOrganization of Visual Features for Rapid Categorization,\u201d Scienti\ufb01c Reports, vol. 8, p. 3752, 12 2018.\n\n[13] S. Grossman, G. Gaziv, E. M. Yeagle, M. Harel, P. Megevand, D. M. Groppe, S. Khuvis, J. L. Herrero,\nM. Irani, A. D. Mehta, and R. Malach, \u201cDeep Convolutional modeling of human face selective columns\nreveals their role in pictorial face representation,\u201d bioRxiv, p. 444323, 10 2018.\n\n[14] M. Eickenberg, A. Gramfort, G. Varoquaux, and B. Thirion, \u201cSeeing it all: Convolutional network layers\n\nmap the function of the human visual system,\u201d NeuroImage, vol. 152, pp. 184\u2013194, 2017.\n\n[15] H. Richard, A. Pinho, B. Thirion, and G. Charpiat, \u201cOptimizing deep video representation to match brain\n\nactivity,\u201d tech. rep., 2018.\n\n[16] T. Horikawa and Y. Kamitani, \u201cGeneric decoding of seen and imagined objects using hierarchical visual\n\nfeatures,\u201d Nature Communications, vol. 8, p. 15037, 10 2015.\n\n[17] S. Zhou, C. R. Cox, and H. Lu, \u201cImproving whole-brain neural decoding of fmri with domain adaptation,\u201d\n\nbioRxiv, 2018.\n\n[18] P. Ghaemmaghami, M. Nabi, Y. Yan, G. Riccardi, and N. Sebe, \u201cA cross-modal adaptation approach for\nbrain decoding,\u201d in 2017 IEEE International Conference on Acoustics, Speech and Signal Processing\n(ICASSP), pp. 969\u2013973, March 2017.\n\n[19] R. Raina, A. Battle, H. Lee, B. Packer, and A. Y. Ng, \u201cSelf-taught learning: Transfer learning from\nunlabeled data,\u201d in Proceedings of the 24th International Conference on Machine Learning, ICML \u201907,\n(New York, NY, USA), pp. 759\u2013766, ACM, 2007.\n\n[20] R. Zhang, P. Isola, and A. A. Efros, \u201cSplit-brain autoencoders: Unsupervised learning by cross-channel\n\nprediction,\u201d in The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017.\n\n[21] A. Tewari, M. Zollhofer, H. Kim, P. Garrido, F. Bernard, P. Perez, and C. Theobalt, \u201cMofa: Model-\nbased deep convolutional face autoencoder for unsupervised monocular reconstruction,\u201d in The IEEE\nInternational Conference on Computer Vision (ICCV) Workshops, Oct 2017.\n\n[22] Y. Pu, Z. Gan, R. Henao, X. Yuan, C. Li, A. Stevens, and L. Carin, \u201cVariational autoencoder for deep\nlearning of images, labels and captions,\u201d in Advances in Neural Information Processing Systems 29 (D. D.\nLee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett, eds.), pp. 2352\u20132360, Curran Associates,\nInc., 2016.\n\n[23] V. VN, \u201cStatistical learning theory,\u201d Wiley, vol. 16, 01 1998.\n\n10\n\n\f[24] R. M. Marcacini, R. G. Rossi, I. P. Matsuno, and S. O. Rezende, \u201cCross-domain aspect extraction for\nsentiment analysis: A transductive learning approach,\u201d Decision Support Systems, vol. 114, pp. 70 \u2013 80,\n2018.\n\n[25] A. Rossi, M. Tiezzi, G. M. Dimitri, M. Bianchini, M. Maggini, and F. Scarselli, \u201cInductive\u2013transductive\nlearning with graph neural networks,\u201d in Arti\ufb01cial Neural Networks in Pattern Recognition (L. Pancioni,\nF. Schwenker, and E. Trentin, eds.), (Cham), pp. 201\u2013212, Springer International Publishing, 2018.\n\n[26] Y. Gao, J. Ma, and A. L. Yuille, \u201cSemi-supervised sparse representation based classi\ufb01cation for face\nrecognition with insuf\ufb01cient labeled samples,\u201d IEEE Transactions on Image Processing, vol. 26, pp. 2545\u2013\n2560, May 2017.\n\n[27] A. Shocher, N. Cohen, and M. Irani, \u201cZero-Shot Super-Resolution using Deep Internal Learning,\u201d 2018.\n[28] Y. Gandelsman, A. Shocher, and M. Irani, \u201c"Double-DIP": Unsupervised Image Decomposition\n\nvia Coupled Deep-Image-Priors,\u201d 12 2018.\n\n[29] A. Shocher, S. Bagon, P. Isola, and M. Irani, \u201cInternal Distribution Matching for Natural Image Retargeting,\u201d\n\n12 2018.\n\n[30] K. Simonyan and A. Zisserman, \u201cVery Deep Convolutional Networks for Large-Scale Image Recognition,\u201d\n\n9 2014.\n\n[31] X. Glorot and Y. Bengio, \u201cUnderstanding the dif\ufb01culty of training deep feedforward neural networks,\u201d 3\n\n2010.\n\n[32] J. Deng, W. Dong, R. Socher, L.-J. Li, Kai Li, and Li Fei-Fei, \u201cImageNet: A large-scale hierarchical image\ndatabase,\u201d in 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248\u2013255, IEEE, 6\n2009.\n\n11\n\n\f", "award": [], "sourceid": 3508, "authors": [{"given_name": "Roman", "family_name": "Beliy", "institution": "weizmann institute"}, {"given_name": "Guy", "family_name": "Gaziv", "institution": "Weizmann Institute of Science"}, {"given_name": "Assaf", "family_name": "Hoogi", "institution": "Weizmann Institute"}, {"given_name": "Francesca", "family_name": "Strappini", "institution": "Weizmann Institute of Science"}, {"given_name": "Tal", "family_name": "Golan", "institution": "Columbia University"}, {"given_name": "Michal", "family_name": "Irani", "institution": "Weizmann Institute of Science"}]}