{"title": "Face Reconstruction from Voice using Generative Adversarial Networks", "book": "Advances in Neural Information Processing Systems", "page_first": 5265, "page_last": 5274, "abstract": "Voice profiling aims at inferring various human parameters from their speech, e.g. gender, age, etc. In this paper, we address the challenge posed by a subtask of voice profiling - reconstructing someone's face from their voice. The task is designed to answer the question: given an audio clip spoken by an unseen person, can we picture a face that has as many common elements, or associations as possible with the speaker, in terms of identity?\n\nTo address this problem, we propose a simple but effective computational framework based on generative adversarial networks (GANs). The network learns to generate faces from voices by matching the identities of generated faces to those of the speakers, on a training set. We evaluate the performance of the network by leveraging a closely related task - cross-modal matching. The results show that our model is able to generate faces that match several biometric characteristics of the speaker, and results in matching accuracies that are much better than chance. The code is publicly available in https://github.com/cmu-mlsp/reconstructing_faces_from_voices", "full_text": "Face Reconstruction from Voice using Generative\n\nAdversarial Networks\n\nYandong Wen\n\nCarnegie Mellon University\n\nPittsburgh, PA 15213\n\nyandongw@andrew.cmu.edu\n\nRita Singh\n\nBhiksha Raj\n\nCarnegie Mellon University\n\nCarnegie Mellon University\n\nPittsburgh, PA 15213\nrsingh@cs.cmu.edu\n\nPittsburgh, PA 15213\nbhiksha@cs.cmu.edu\n\nAbstract\n\nVoice pro\ufb01ling aims at inferring various human parameters from their speech, e.g.\ngender, age, etc. In this paper, we address the challenge posed by a subtask of voice\npro\ufb01ling - reconstructing someone\u2019s face from their voice. The task is designed\nto answer the question: given an audio clip spoken by an unseen person, can we\npicture a face that has as many common elements, or associations as possible with\nthe speaker, in terms of identity?\nTo address this problem, we propose a simple but effective computational\nframework based on generative adversarial networks (GANs). The network\nlearns to generate faces from voices by matching the identities of generated\nfaces to those of the speakers, on a training set. We evaluate the perfor-\nmance of the network by leveraging a closely related task - cross-modal match-\ning. The results show that our model is able to generate faces that match\nseveral biometric characteristics of the speaker, and results in matching accu-\nracies that are much better than chance. The code is publicly available in\nhttps://github.com/cmu-mlsp/reconstructing_faces_from_voices\n\n1\n\nIntroduction\n\nThe challenge of voice pro\ufb01ling is to infer a person\u2019s biophysical parameters, such as gender, age,\nhealth conditions, etc. from their speech, and a large body of literature exists on the topic [28, 1, 21].\nIn this paper, we extend this challenge and address the problem: is it possible to go beyond merely\npredicting a person\u2019s physical attributes, and actually reconstruct their entire face from their voice?\nEffectively a new subtask of voice pro\ufb01ling, the task is designed to answer the question: given an\nunheard audio clip spoken by an unseen person, can we picture a face image that has as many as\npossible associations with the speaker in terms of identity?\nA person\u2019s voice is incontrovertibly statistically related to their facial structure. The relationship\nis, in fact, multi-faceted. Direct relationships include the effect of the underlying skeletal and\narticulator structure of the face and the tissue covering them, all of which govern the shapes, sizes,\nand acoustic properties of the vocal tract that produces the voice [22, 38]. Less directly, the same\ngenetic, physical and environmental in\ufb02uences that affect the development of the face also affect the\nvoice. Demographic factors, such as gender, age and ethnicity too in\ufb02uence both voice and face (and\ncan in fact be independently inferred from the voice [1, 15] or the face [17]), providing additional\nlinks between the two.\nNeurocognitive studies have shown that human perception implicitly recognizes the association\nof faces to voices [4]. Studies indicate that neuro-cognitive pathways for voices share a common\nstructure with that for faces [7] \u2013 the two may follow parallel pathways within a common recognition\nframework [4, 3]. In empirical studies humans have shown the ability to associate voices of unknown\nindividuals to pictures of their faces [11, 19]. They are seen to show improved ability to memorize\n\n33rd Conference on Neural Information Processing Systems (NeurIPS 2019), Vancouver, Canada.\n\n\fFigure 1: The proposed GANs-based framework for generating faces from voices.\ncomponents: voice embedding network, generator, discriminator, and classi\ufb01er.\n\nIt includes 4 major\n\nand recall voices when previously shown pictures of the speaker\u2019s face, but not imposter faces\n[20, 32, 31].\nGiven these demonstrable dependencies, it is reasonable to hypothesize that it may also be possible\nto reconstruct face images from a voice signal algorithmically.\nOn the other hand, reconstructing the face from voice is a challenging, maybe even impossible task\nfor several reasons [35]. First, it is an ill-posed cross-modal problem : although many face-related\nfactors affect the voice, it may not be possible to entirely disambiguate them from the voice. Even if\nthis were not the case, it is unknown a priori exactly what features of the voice encode information\nabout any given facial feature (although one may take guesses [35]). Moreover, the signatures of the\ndifferent facial characteristics may lie in different spoken sounds; thus, in order to obtain suf\ufb01cient\nevidence, the voice recordings must be long enough to have suf\ufb01cient coverage of sounds to derive\nall the necessary information. The information containing in a single audio clip may not be suf\ufb01cient\nfor constructing a face image.\nYet, although prima facie the problem seems extremely hard, recent advances in neural network based\ngenerative models have shown that they are able to perform similarly challenging generative tasks\nin a variety of scenarios, when properly structured and trained. In particular, generative adversarial\nnetworks (GANs) have demonstrated the ability to learn to generate highly sophisticated imagery,\ngiven only signals about the validity of the generated image, rather than detailed supervision of the\ncontent of the image itself [23, 30, 40]. We use this ability to learn to generate faces from voices.\nFor our solution, we propose a simple but effective data-driven framework based on generative\nadversarial networks (GANs), as illustrated in Fig. 1. The objective of the network is simple: given a\nvoice recording it must generate a face image that plausibly belongs to that voice. The voice recording\nitself is input to the generator network in the form of a voice embedding vector extracted by a voice\nembedding network. The generator is trained using a pair of discriminators. The \ufb01rst evaluates if the\nimages it generates are realistic face images. The second discriminator (classi\ufb01er) veri\ufb01es that the\nidentity of face image output by the generator does indeed match the actual identity of the speaker.\nWe present both qualitative and quantitative evaluations of the results produced by our model. The\nqualitative results show that our framework is able to map the voice manifold to face manifold.\nWe can observe many identity associations between the generated faces and the input voices. The\ngenerated faces are generally age and gender appropriate, frequently matching the real face of the\nspeaker. Additionally, given non-speech input the outputs become unrealistic, showing that the\nlearned mapping is at least somewhat speci\ufb01c, in that the face manifold it learns are derived primarily\nfrom the voice manifold and not elsewhere. In addition, for different speech segments from the same\nperson, the generated faces exhibit reasonable intra-class variation.\nWe also propose a number of quantitative evaluation metrics to evaluate the output of our network,\nbased on how speci\ufb01c the model is in mapping voices to faces, how well the high-level attributes of\nthe generated face match that of the speaker, and how well the generated faces match the identity (ID)\nof the speaker itself. For the last metric (ID matching), we leverage the cross-modal matching task\n[25], wherein, speci\ufb01cally, we need to match a speech segment to one of the two faces, where one is\nthe true face of the speaker, and another is an \u201cimposter.\u201d Our tests reveal that the network is highly\nspeci\ufb01c in generating faces in response to voices, produces quanti\ufb01ably gender-appropriate faces\nfrom voices, and that the matching accuracy is much better than chance, or what may be obtained\nmerely by matching gender. We refer the reader to the experiments section for actual numbers.\n\n2\n\nVoice Embedding NetworkVoice Embedding NetworkGeneratorGeneratorGenerated FaceDiscriminatorDiscriminatorClassifierClassifierDiscriminatorClassifierVoice EmbeddingVoice Recording\fOverall, our contributions are summarized as follows:\n\nto explore the relationship between voice and face modalities.\n\n\u2022 We introduce a new task of generating faces from voice in voice pro\ufb01ling. It could be used\n\u2022 We propose a simple but effective framework based on generative adversarial networks for\n\u2022 We propose to quantitatively evaluate the generated faces by using a cross-modal matching\ntask. Both the qualitative and quantitative results show that our framework is able to generate\nfaces that have identity association with the input voice.\n\nthis task. Each component in the framework is well motivated.\n\n2 Related Works\n\nThe task of deriving faces from voices relates to several research areas.\nVoice pro\ufb01ling. There currently exists a signi\ufb01cant body of research on deriving personal pro\ufb01le\nparameters from a person\u2019s voice [35]. Many useful characteristics about the speaker can be inferred\nfrom their speech, e.g. age [28, 1], emotion [21], identity [6], anthropometric measurement [36],\nhealth status [33] etc. These pro\ufb01le parameters may be viewed as providing linkages between a\nperson\u2019s voice and their face.\nFace generation using GANs. There is a long line of research on generating faces using GANs.\n[29, 12] can be used to generate face images from noise, but it does not consider any conditioning\ninformation (like identity) as input. [27, 18] use a discrete label as the condition, but it only works\non a closed-set scenario. [8, 2] achieve identity-preserving face generation in an open-set scenario.\nHowever, the problem they focus on is within-modal generation problem, where a reference face\nimage of the target identity to be generated is shown to the model. Our task is more challenging,\nsince the conditioning information provided is actually from a different modality, namely voice.\nVoice to face matching. Cross modal matching between voice and face has become an increasingly\npopular problem in recent years. It aims at matching a probe input from one modality to a gallery\nof multiple inputs from the other modality. [25] formulates this task as a N-way classi\ufb01cation\nproblem. [24, 13, 39] propose to learn common embeddings for the cross-modal inputs, such that\nthe matching can be performed using the learned embeddings. All of these, however, are essentially\nselection problems. Our task requires actual generation is naturally more challenging than cross-\nmodal matching. In part, this is also because many possible generated images could be the expected\noutput for an audio input. There is no unique target output as a supervision signal to train our model.\nTalking face. Talking face [41, 10] is a recently proposed task. Given a static face image and a\nspeech clip, the goal is to generate a sequence of target face images where the lips are synchronized\nwith the audio. The talking face task is signi\ufb01cantly different from our task; in fact the two are\nactually somewhat orthogonal to each other because talking face focuses on the content of the speech\nand ignores the identity of the speaker, while our task does the opposite, extracting the identity of\nspeaker and discarding the content of speech.\nSpeech to face. A very recent, yet to be presented publication that has been brought to our notice by\nthe authors also reports on the production of faces from voices [37]. While the problem addressed\nis similar, the approach claimed is different. Since the codebase and much of the necessary detail\nremain unpublished at the time of this submission1, we do not perform comparative assessments.\n\n3 The Proposed Framework\n\nBefore we begin, we \ufb01rst specify some of the notation we will use. We represent voice recordings\nby the symbol v, using super or subscripts to identify speci\ufb01c recordings. Similarly, we represent\nface images by the symbol f. We will represent the identity of a subject who provides voice or face\ndata as y. We will represent the true identity of (the subject of) voice recording v as yv and face\nf as yf . We represent the function that maps a voice or face recording to its idenitity as ID(), i.e.\nyv = ID(v) and yf = ID(f ). Additional notation will become apparent as we introduce it.\n\n1A version of the work reported in this paper was tested live by nearly 1000 people at the World Economic\n\nForum in Tianjin, Sep 2018.\n\n3\n\n\fOur objective is to train a model F (v; \u0398) (with parameter \u0398) that takes as input a voice recording\nv and produces, as output a face image \u02c6f = F (v; \u0398) that belongs to the speaker of v, i.e. such that\nID( \u02c6f ) = ID(v).\nWe use the framework shown in Figure 1 for our model, which decomposes F (v; \u0398) into a sequence\nof two components, Fe(v; \u03b8e) and Fg(e; \u03b8g). Fe(v; \u03b8e) : v \u2192 e is a voice embedding function with\nparameter \u03b8e that takes in a voice recording v and outputs an embedding vector e that captures all the\nsalient information in v. Fg(e; \u03b8g) : e \u2192 f is a generator function that takes in an embedding vector\nand generates a face image \u02c6f.\nWe must learn \u03b8e and \u03b8g such that ID(Fg(Fv(v; \u03b8e); \u03b8g) = ID(v).\n\n3.1 Training the network\n\n3.1.1 Data\nWe assume the availability of face and voice data from a set of subjects Y = {y1, y2,\u00b7\u00b7\u00b7 , yk}.\nCorrespondingly, we also have a set of voice recordings V = {v1, v2, ..., vN}, with identity\nlabels Y v = {yv\nN} and a set of faces F = {f1, f2, ..., fM} with identity labels\n1 , yv\n2 , ..., yv\nY f = {yf\nM}, such that yv \u2208 Y \u2200yv \u2208 Y v and yf \u2208 Y \u2200yf \u2208 Y f . N may not be\n2 , ..., yf\nequal to M.\nIn addition, we de\ufb01ne two sets of labels R = {r1, r2, ..., rM | \u2200i, ri = 1} and \u02c6R = {\u02c6r1, \u02c6r2, ..., \u02c6rN |\n\u2200i, \u02c6ri = 0} corresponding Y f and Y v respectively. R is a set of labels that indicates that all faces in\nF are \u201creal.\u201d \u02c6R is a set of labels that indicates that any faces generated from any v \u2208 V are synthetic\nor \u201cfake.\u201d\n\n1 , yf\n\n3.1.2 GAN framework\nIn training the model, we impose two requirements. First, the output \u02c6f of the generator in response to\nany actual voice input v must be a realistic face image. Second, it must belong to the same identity\nas the voice, i.e. ID( \u02c6f ) = f v. As explained in Section 1, we will use a GAN framework to train\nFe(.; \u03b8e) and Fg(.; \u03b8g). This will require the de\ufb01nition of adversary that provide losses that can be\nused to learn the model parameters.\nWe de\ufb01ne an adversarial objective. First, the discriminator Fd determines if any input image (f or \u02c6f)\nis a genuine picture of a face, or one generated by the generator, i.e. assigns any face image (f or \u02c6f)\nto its real/fake label (r or \u02c6r). The loss function for Fd is de\ufb01ned as Ld(Fd(f ), r) (or Ld(Fd( \u02c6f ), \u02c6r)).\nSecond, classi\ufb01er Fc learns to assign any real face image f to its identity label yf . Accordingly, the\nloss function for Fc is Lc(Fc(f ), yf ). Last, the generator Fg takes in a voice recording v and attempts\nto generate any face image \u02c6f that can be classi\ufb01ed to real label r and identity label yv by Fd and Fg,\nrespectively. The corresponding loss function for Fg is Ld(Fd(Fg(v), r) + Lc(Fc(Fg(v)), yv).\nIn our implementation, we instantiate Fe(v; \u03b8e), Fg(e; \u03b8g), Fd(f ; \u03b8d) and Fc(f ; \u03b8c) as convolutional\nneural networks, shown in Fig. 1. Fe is the component labeled as Voice Embedding Network. v is the\nMel-Spectrographic representations of speech signal. The output of the \ufb01nal convolutional layer is\npooled over time, leading to a q-dimensional vector e. Fg is labeled as Generator. f and \u02c6f are RGB\nimages with the same resolution of w \u00d7 h. Fd and Fc are labeled as Discriminator and Classi\ufb01er,\nrespectively. The loss functions Ld and Lc of these two components are the cross-entropy loss.\n\n3.1.3 Training the network\nThe training data comprise a set of voice recordings V and a set of face images F. From the voice\nrecordings in V we could obtain the corresponding generated face images \u02c6F = { \u02c6f = Fg(Fe(v)) |\n\u2200v \u2208 V}.\nThe framework is trained in an adversarial manner. To simplify, we use a pretrained voice embedding\nnetwork Fe(v; \u03b8e) from a speaker recognition task, and freeze the parameter \u03b8e when training our\ni=1 Ld(Fd( \u02c6fi), \u02c6ri) with \ufb01xed \u03b8e, \u03b8g\ni=1 Lc(Fc(fi), yi) with \ufb01xed \u03b8e, \u03b8g and \u03b8d. The\n\nframework. Fd is trained to maximize(cid:80)M\nand \u03b8c. Similarly, the Fc is trained to maximize(cid:80)M\n\ni=1 Ld(Fd(fi), ri) +(cid:80)N\n\n4\n\n\fFg is trained to maximize(cid:80)N\n\n\u02c6fi = Fg(Fe(vi)). The training pipeline is summarized in Algorithm 1.\n\ni=1 Ld(Fd( \u02c6fi), ri) + Lc(Fc( \u02c6fi), yv\n\ni ) with \u03b8e, \u03b8d and \u03b8c \ufb01xed, where\n\nAlgorithm 1 The training algorithm of the proposed framework\nInput: A set of voice recordings with identity label (V,Y v). A set of labeled face images with identity label\n(F,Y f ). A voice embedding network Fe(v; \u03b8e) trained on V with speaker recognition task. \u03b8e is \ufb01xed\nduring the training. Randomly initialized \u03b8g, \u03b8d, and \u03b8c\n\nOutput: The parameters \u03b8g.\n1: while not converge do\n2:\n3:\n4:\n\nRandomly sample a minibatch of n voice recordings {v1, v2, ..., vn} from V\nRandomly sample a minibatch of m face images {f1, f2, ..., fm} from F\nUpdate the discriminator Fd(f ; \u03b8d) by ascending the gradient\n\n(cid:0)(cid:80)n\ni=1 log(1 \u2212 Fd( \u02c6fi)) +(cid:80)m\n(cid:0)(cid:80)m\n(cid:0)(cid:80)n\n\ni=1 log Fc(fi)[yf\n\ni=1 log Fc(Fg(Fe(vi)))[yv\n\ni=1 log Fd(fi)(cid:1)\ni ] +(cid:80)m\n\ni ](cid:1)\n\n\u2207\u03b8d\n\u2207\u03b8c\n\u2207\u03b8g\n\nUpdate the generator Fg(f ; \u03b8c) by ascending the gradient\n\ni=1 log Fd(Fg(Fe(vi)))(cid:1)\n\nUpdate the classi\ufb01er Fc(f ; \u03b8c) by ascending the gradient (a[i] indicates the i-th element of vector a)\n\n5:\n\n6:\n\n7: end while\n\nOnce trained, Fd(f ; \u03b8d) and Fc(f ; \u03b8c) can be removed. Only Fd(f ; \u03b8d) and Fc(f ; \u03b8c) are used for\nface generation from voice in the inference. It is worth noting that the targeted scenario is open-set.\nIn our evaluations, the model is required to work on previously unseen and unheard identities, i.e.\nyv \u2208 Y in the training phase, while yv /\u2208 Y in the testing phase.\n\n4 Experiments\n\nIn our experiments, the voice recordings are from the Voxceleb [25] dataset and the face images\nare from the manually \ufb01ltered version of VGGFace [26] dataset. Both datasets have identity labels.\nWe use the intersection of the two datasets with the common identities, leading to 149,354 voice\nrecordings and 139,572 frontal face images of 1,225 subjects. We follow the train/validation/test split\nin [25]. The details are shown in Table 1.\nSeparated data pre-processing pipelines are employed to audio segments and face images. For audio\nsegments, we use a voice activity detector interface from the WebRTC project to isolate speech-\nbearing regions of the recordings. Subsequently, we extract 64-dimensional log mel-spectrograms\nusing an analysis window of 25ms, with a hop of 10ms between frames. We perform mean and\nvariance normalization of each mel-frequency bin. We randomly crop an audio clips around 3 to 8\nseconds for training, but use the entire recording for testing. For the face data, facial landmarks in all\nimages are detected using [5]. The cropped RGB face images of size 3 \u00d7 64 \u00d7 64 are obtained by\nsimilarity transformation. Each pixel in the RGB images is normalized by subtracting 127.5 and then\ndividing by 127.5.\n\nTable 1: Statistics of the datasets used in our experiments\n\n# of speech segments\n# of face images\n# of subjects\n\ntrain\n\n113,322\n106,584\n\n924\n\nvalidation\n\n14,182\n12,533\n\n112\n\ntest\n\n21,850\n20,455\n\n189\n\ntotal\n\n149,354\n139572\n1,225\n\nTraining. The network architecture is given in Table 2. The parameters in the convolutional layers of\ndiscriminator and classi\ufb01er are shared in our experiments. We basically follow the hyperparameter\nsetting in [29]. We used the Adam optimizer [14] with learning rate of 0.0002. \u03b21 and \u03b22 are 0.5 and\n0.999, respectively. Minibatch size is 128. The training is completed at 100K iterations.\n\n4.1 Qualitative Results\nAs a \ufb01rst experiment, we compared the outputs of the network in response to various noise signals to\noutputs obtained from actual speech recordings. Figure 2 shows outputs generated for four different\ntypes of noise. We evaluated noise segments of different durations (1,2,3, 5 and 10 seconds) to\nobserve how the generated faces change with the duration. The generated images are seen to be\n\n5\n\n\fTable 2: The detailed CNNs architectures. For the voice embedding network, we use 1D convolutional layers.\nConv 3/2,1 denotes 1D convoluitonal layer with kernel size of 3, where the stride and padding are 2 and 1,\nrespectively. Each convolutional layer is followed by a Batch Normalization (BN) [9] layer and Recti\ufb01ed Linear\nUnits (ReLU) [16]. The output shape is shown accordingly, where ti+1 = (cid:100)(ti \u2212 1)/2(cid:101) + 1. The \ufb01nal outputs\nare pooled over time, yielding a 64-dimensional embedding. We use 2D deconvolutional layers with ReLU for\nthe generator and 2D convolutional layers with Leaky ReLU (LReLU) for the discriminator and classi\ufb01er. The\n\ufb01nal output is given by fully connected (FC) layer.\n\nVoice Embedding Network\n\nLayer\nInput\n\nConv 3/2,1\nConv 3/2,1\nConv 3/2,1\nConv 3/2,1\nConv 3/2,1\n\nAvePool 1 \u00d7 t5\n\nLayer\nInput\n\nConv 1 \u00d7 1/1,0\nConv 3 \u00d7 3/2,1\nConv 3 \u00d7 3/2,1\nConv 3 \u00d7 3/2,1\nConv 3 \u00d7 3/2,1\nConv 4 \u00d7 4/1,0\n\nFC 64 \u00d7 1\n\nAct.\n\n-\n\nBN + ReLU\nBN + ReLU\nBN + ReLU\nBN + ReLU\nBN + ReLU\n\nDiscriminator\n\nAct.\n\n-\n\n-\n\nLReLU\nLReLU\nLReLU\nLReLU\nLReLU\nLReLU\nSigmoid\n\nOutput shape\n64 \u00d7 t0\n256 \u00d7 t1\n384 \u00d7 t2\n576 \u00d7 t3\n864 \u00d7 t4\n64 \u00d7 t5\n64\u00d71\n\nOutput shape\n3 \u00d7 64 \u00d7 64\n32 \u00d7 64 \u00d7 64\n64 \u00d7 32 \u00d7 32\n128 \u00d7 16 \u00d7 16\n256 \u00d7 8 \u00d7 8\n512 \u00d7 4 \u00d7 4\n64 \u00d7 1 \u00d7 1\n\n1\n\nLayer\nInput\n\nDeconv 4 \u00d7 4/1,0\nDeconv 3 \u00d7 3/2,1\nDeconv 3 \u00d7 3/2,1\nDeconv 3 \u00d7 3/2,1\nDeconv 3 \u00d7 3/2,1\nDeconv 1 \u00d7 1/1,0\n\nLayer\nInput\n\nConv 1 \u00d7 1/1,0\nConv 3 \u00d7 3/2,1\nConv 3 \u00d7 3/2,1\nConv 3 \u00d7 3/2,1\nConv 3 \u00d7 3/2,1\nConv 4 \u00d7 4/1,0\n\nFC 64 \u00d7 k\n\nGenerator\n\nAct.\n\n-\n\nReLU\nReLU\nReLU\nReLU\nReLU\n\nClassi\ufb01er\nAct.\n\n-\n\n-\n\nLReLU\nLReLU\nLReLU\nLReLU\nLReLU\nLReLU\nSoftmax\n\nOutput shape\n64 \u00d7 1 \u00d7 1\n1024 \u00d7 4 \u00d7 4\n512 \u00d7 8 \u00d7 8\n256 \u00d7 16 \u00d7 16\n128 \u00d7 32 \u00d7 32\n64 \u00d7 64 \u00d7 64\n3 \u00d7 64 \u00d7 64\n\nOutput shape\n3 \u00d7 64 \u00d7 64\n32 \u00d7 64 \u00d7 64\n64 \u00d7 32 \u00d7 32\n128 \u00d7 16 \u00d7 16\n256 \u00d7 8 \u00d7 8\n512 \u00d7 4 \u00d7 4\n64 \u00d7 1 \u00d7 1\n\nk\n\nFigure 2: The generated face images from noise input. Each row shows the generated faces using one of the\nfour noise audio segments with different durations. (a) 1 second, (b) 2 seconds, (c) 3 seconds, (d) 5 seconds, (e)\n10 seconds.\n\nblurry, unrecognizable and generally alike, since there is no identity information in noise. With longer\nnoise recordings, the results do not improve. Similar results are obtained over a variety of noises.\nOn the other hand, when we use regular speech recordings with the aforementioned durations as\ninputs, outputs tend to be realistic faces, as seen in Figure 3. The results indicate that while the\ngenerator does learn to produce face-like images, actual faces are produced chie\ufb02y in response to\nactual voice. We infer that while the generator has learned to map the speech manifold to the manifold\nof faces, it maps other inputs outside of this manifold.\nFigure 3 also enables us to subjectively evaluate the actual output of the network. These results\nare typical and not cherry-picked for presentation (several of our reconstructions match the actual\nspeaker closely, but we have chosen not to selectively present those to avoid misrepresenting the\nactual performance of the system). The generated images are on the left, while the reference images\n(the actual faces of the speakers) are on the right. To reduce the perceptual bias and better illustration,\nwe show multiple reference face images for each speaker. Although the generated and the reference\n\n6\n\n(a) 1s(b) 2s(d) 5s(e) 10s(c) 3sWhite Gaussian noiseBabble noisePink noiseBrown noise\fFigure 3: (a)-(e) The generated face images from regular speech recordings with different durations. (f) the\ncorresponding reference face images. These 4 speakers (from top to bottom) are Danica McKellar, Cindy\nWilliams, Damian Lewis, and Eva Green.\n\nface indicate different persons, the identity information of these two are matched in some sense (like\ngender, ethnicity, etc.). With longer speech segments, the generated faces gradually converge to faces\nassociatable with the speaker. Figure 4 shows additional examples demonstrating that the synthesized\nimages are generally age- and gender-matched with the speaker.\n\nFigure 4: (a) Faces from old voices. (b) Faces from male voices. (c) Faces from female voices. For each group,\nimages on the left are the generated images and images on the right are the references.\n\nFigure 5: The generated face images from different speech segments of the same speaker and their corresponding\nface images. Each row shows the results from the same speaker.\n\nIn the next experiment, we select 7 different speech segments of each speaker and generate the faces.\nEntire recordings are used. The results are shown in Figure 5 (reference images are also provided for\ncomparison). Once again, the results are typical and not cherrypicked. We believe that the images in\neach row exhibit reasonable variations of the same person (except the fourth image in the \ufb01rst row),\nindicating that our model is able to build a mapping between the speech group to face group, thus\nretaining the identity of the face across speech segments from the same speaker.\n\n7\n\n(a) 1s(b) 2s(d) 5s(e) 10s(c) 3s(f) Reference images(a)(b)(c)(a) Generated face images(b) Reference images\f4.2 Quantitative Results\n\nWe attempt to quantitatively distinguish between the faces generated in response to noise from those\ngenerated from voice using the discriminator Fd() itself. Note that Fd() is biased, and has explicitly\nbeen trained to tag synthesized faces as fake. The mean and standard deviation (obtained from 1000\nsamples) of its output value in response to actual voices are 0.09 and 0.13 respectively, while for (an\nidentical number of) noise inputs they are 0.06 and 0.07 respectively. Even a biased discriminator is\nclearly able to distinguish between faces generated from voices, and those obtained from noise.\nAs a \ufb01rst test, we attempted to ID (classify) faces derived from test recordings of the speakers in\nthe training set. On a set of 4620 recordings, from 924 identities, we achieved a top-1 accuracy of\n61.7% and a top-5 accuracy of 82.3%, showing that the voice-based face reconstructions do actually\nfaithfully capture the identities of known subjects.\nFor unknown subjects, we ran a gender classi\ufb01er on 21,850 generated faces (from the speech\nsegments in test set). The classi\ufb01er was trained on the face images in the training set using the\nnetwork architecture of the discriminator, with an accuracy of 98.97% on real face images. The\nclassi\ufb01er obtained a 96.45% accuracy in matching the gender of the generated faces to the known\ngender of the speaker, showing that the generated faces are almost always of the correct gender.\nFinally, we evaluated our model by leveraging the task of voice to face matching. Here, we are given\na voice recording, a face image of the true speaker, and a face image of an imposter. We must match\nthe voice to the face of the true speaker. Ideally, the probe voice could be replaced by the generated\nface image if they carry the same identity information. So the voice to face matching problem reduces\nto a typical face veri\ufb01cation or face recognition problem. The resulting matching accuracy could be\nused to quantitatively evaluate the association between the speech segment and the generated face.\nWe construct the testing instances (a probe voice recording, a true face image, and an \u201cimposter\u201d\nface) using data in the testing set, leading to 2,353,560 trials. We also compute the matching accuracy\non about 50k trials constructed from a small part of the training set to see how well the model \ufb01t\nto the training data. We also perform strati\ufb01ed experiments based on gender where we select the\nimposter face with the same gender as the true face. In this case, gender information cannot be used\nfor matching anymore, leading to a more fair test.\nThe results are shown in Table 3. The high accuracies obtained on the training set for the unstrati\ufb01ed\nand gender strati\ufb01ed tests (96.83% and 93.98% respectively) show that generated faces do carry\ncorrect identity information for the training set. The results on the test set on unstrati\ufb01ed and gender\nstrati\ufb01ed tests (76.07% and 59.69% respectively) are better than those in DIMNets-G [39], indicating\nthat our model learns more associations than gender. The large drop compared to the results on the\ntraining set shows however that considerable room remains to improve generalizability in the model.\n\nTable 3: The voice to face matching accuracies. Our results are given by replacing the probe voice embeddings\nby the embeddings of the generated face.\n\nSVHF [25]\n\nDIMNets-I [39]\nDIMNets-G [39]\n\nours\n\nunstrati\ufb01ed group (ACC. %)\n\n(training set / testing set)\n\n- / 81.00\n- / 83.45\n- / 72.90\n\n96.83 / 76.07\n\nstrati\ufb01ed group by gender (ACC. %)\n\n(training set / testing set)\n\n- / 65.20\n- / 70.91\n- / 50.32\n\n93.98 / 59.69\n\n5 Discussion and Conclusion\n\nThe proposed GAN-based framework is seen to achieve reasonable reconstruction results. The\ngenerated faces have identity associations with the true speaker. There remains considerable room\nfor improvement. Firstly, there are obvious issues with the GAN-based output: the produced faces\nhave features such as hair that are presumably not predicted by voice, but simply obtained from\ntheir co-occurrence with other features. The model may be more appropriately learned through data\ncleaning that removes obviously unrelated aspects of the facial image, such as hair and background.\nThe proposed model is vanilla in many ways. For instance [34, 36] describe several explicit corre-\nspondences between speech and face features, e.g. different phonetic units are known to relate to\ndifferent facial features. We are investigating models that explicitly consider these issues.\n\n8\n\n\fReferences\n[1] Mohamad Hasan Bahari, ML McLaren, DA Van Leeuwen, et al. Age estimation from telephone speech\n\nusing i-vectors. 2012.\n\n[2] Jianmin Bao, Dong Chen, Fang Wen, Houqiang Li, and Gang Hua. Towards open-set identity preserving\nface synthesis. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages\n6713\u20136722, 2018.\n\n[3] P. Belin, P.E.G. Bestelmeyer, M. Latinus, and R. Watson. Understanding voice perception. British Journal\n\nof Psychology, 108:711\u2013725, 2011.\n\n[4] Pascal Belin, Shirley Fecteau, and Catherine Bedard. Thinking the voice: neural correlates of voice\n\nperception. Trends in cognitive sciences, 8(3):129\u2013135, 2004.\n\n[5] Adrian Bulat and Georgios Tzimiropoulos. How far are we from solving the 2d & 3d face alignment\nproblem? (and a dataset of 230,000 3d facial landmarks). In International Conference on Computer Vision,\n2017.\n\n[6] Najim Dehak, Patrick J Kenny, R\u00e9da Dehak, Pierre Dumouchel, and Pierre Ouellet. Front-end fac-\ntor analysis for speaker veri\ufb01cation. IEEE Transactions on Audio, Speech, and Language Processing,\n19(4):788\u2013798, 2010.\n\n[7] Andrew W Ellis. Neuro-cognitive processing of faces and voices. In Handbook of research on face\n\nprocessing, pages 207\u2013215. Elsevier, 1989.\n\n[8] Rui Huang, Shu Zhang, Tianyu Li, and Ran He. Beyond face rotation: Global and local perception gan for\nphotorealistic and identity preserving frontal view synthesis. In Proceedings of the IEEE International\nConference on Computer Vision, pages 2439\u20132448, 2017.\n\n[9] Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing\n\ninternal covariate shift. arXiv preprint arXiv:1502.03167, 2015.\n\n[10] Amir Jamaludin, Joon Son Chung, and Andrew Zisserman. You said that?: Synthesising talking faces from\n\naudio. International Journal of Computer Vision, pages 1\u201313, 2019.\n\n[11] Miyuki Kamachi, Harold Hill, Karen Lander, and Eric Vatikiotis-Bateson. Putting the face to the voice\u2019:\n\nMatching identity across modality. Current Biology, 13(19):1709\u20131714, 2003.\n\n[12] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved\n\nquality, stability, and variation. arXiv preprint arXiv:1710.10196, 2017.\n\n[13] Changil Kim, Hijung Valentina Shin, Tae-Hyun Oh, Alexandre Kaspar, Mohamed Elgharib, and Wojciech\n\nMatusik. On learning associations of faces and voices. arXiv preprint arXiv:1805.05553, 2018.\n\n[14] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint\n\narXiv:1412.6980, 2014.\n\n[15] Margarita Kotti and Constantine Kotropoulos. Gender classi\ufb01cation in two emotional speech databases. In\n\n2008 19th International Conference on Pattern Recognition, pages 1\u20134. IEEE, 2008.\n\n[16] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classi\ufb01cation with deep convolutional\n\nneural networks. In Advances in neural information processing systems, pages 1097\u20131105, 2012.\n\n[17] Neeraj Kumar, Alexander C Berg, Peter N Belhumeur, and Shree K Nayar. Attribute and simile classi\ufb01ers\nfor face veri\ufb01cation. In 2009 IEEE 12th International Conference on Computer Vision, pages 365\u2013372.\nIEEE, 2009.\n\n[18] Yongyi Lu, Yu-Wing Tai, and Chi-Keung Tang. Conditional cyclegan for attribute guided face image\n\ngeneration. arXiv preprint arXiv:1705.09966, 2017.\n\n[19] Lauren W Mavica and Elan Barenholtz. Matching voice and face identity from static images. Journal of\n\nExperimental Psychology: Human Perception and Performance, 39(2):307, 2013.\n\n[20] Hunter A McAllister, Robert HI Dale, Norman J Bregman, Allyssa McCabe, and C Randy Cotton. When\neyewitnesses are also earwitnesses: Effects on visual and voice identi\ufb01cations. Basic and Applied Social\nPsychology, 14(2):161\u2013170, 1993.\n\n9\n\n\f[21] Sin\u00e9ad McGilloway, Roddy Cowie, Ellen Douglas-Cowie, Stan Gielen, Machiel Westerdijk, and Sybert\nStroeve. Approaching automatic recognition of emotion from voice: A rough benchmark. In ISCA Tutorial\nand Research Workshop (ITRW) on Speech and Emotion, 2000.\n\n[22] Paul Mermelstein. Determination of the vocal-tract shape from measured formant frequencies. The Journal\n\nof the Acoustical Society of America, 41(5):1283\u20131294, 1967.\n\n[23] Mehdi Mirza and Simon Osindero. Conditional generative adversarial nets. arXiv preprint arXiv:1411.1784,\n\n2014.\n\n[24] Arsha Nagrani, Samuel Albanie, and Andrew Zisserman. Learnable pins: Cross-modal embeddings for\n\nperson identity. arXiv preprint arXiv:1805.00833, 2018.\n\n[25] Arsha Nagrani, Samuel Albanie, and Andrew Zisserman. Seeing voices and hearing faces: Cross-modal\nbiometric matching. In Proceedings of the IEEE conference on computer vision and pattern recognition,\npages 8427\u20138436, 2018.\n\n[26] Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, et al. Deep face recognition. In bmvc, volume 1,\n\npage 6, 2015.\n\n[27] Guim Perarnau, Joost Van De Weijer, Bogdan Raducanu, and Jose M \u00c1lvarez. Invertible conditional gans\n\nfor image editing. arXiv preprint arXiv:1611.06355, 2016.\n\n[28] Paul H Ptacek and Eric K Sander. Age recognition from voice. Journal of speech and hearing Research,\n\n9(2):273\u2013277, 1966.\n\n[29] Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep\n\nconvolutional generative adversarial networks. arXiv preprint arXiv:1511.06434, 2015.\n\n[30] Scott Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Bernt Schiele, and Honglak Lee.\n\nGenerative adversarial text to image synthesis. arXiv preprint arXiv:1605.05396, 2016.\n\n[31] Stefan R Schweinberger, Nadine Kloth, and David MC Robertson. Hearing facial identities: Brain\n\ncorrelates of face\u2013voice integration in person identi\ufb01cation. Cortex, 47(9):1026\u20131037, 2011.\n\n[32] Stefan R Schweinberger, David Robertson, and J\u00fcrgen M Kaufmann. Hearing facial identities. Quarterly\n\nJournal of Experimental Psychology, 60(10):1446\u20131456, 2007.\n\n[33] Jahanzeb Sherwani, Nosheen Ali, Sarwat Mirza, Anjum Fatma, Yousuf Memon, Mehtab Karim, Rahul\nTongia, and Roni Rosenfeld. Healthline: Speech-based access to health information by low-literate users.\nIn 2007 International Conference on Information and Communication Technologies and Development,\npages 1\u20139. IEEE, 2007.\n\n[34] Rita Singh. Pro\ufb01ling humans from their voice. Springer, 2019.\n\n[35] Rita Singh. Reconstruction of the human persona in 3d, and its reverse. In Pro\ufb01ling Humans from their\n\nVoice, chapter 10. springer nature Press, 2020.\n\n[36] Rita Singh, Bhiksha Raj, and Deniz Gencaga. Forensic anthropometry from voice: an articulatory-\nphonetic approach. In 2016 39th International Convention on Information and Communication Technology,\nElectronics and Microelectronics (MIPRO), pages 1375\u20131380. IEEE, 2016.\n\n[37] Changil Kim Inbar Mosseri William T. Freeman Michael Rubinstein Wojciech Matusik Tae-Hyun Oh,\nTali Dekel*. Speech2face: Learning the face behind a voice. In Proceedings of the IEEE conference on\ncomputer vision and pattern recognition, 2019.\n\n[38] HM Teager and SM Teager. Evidence for nonlinear sound production mechanisms in the vocal tract. In\n\nSpeech production and speech modelling, pages 241\u2013261. Springer, 1990.\n\n[39] Yandong Wen, Mahmoud Al Ismail, Weiyang Liu, Bhiksha Raj, and Rita Singh. Disjoint mapping network\n\nfor cross-modal matching of voices and faces. arXiv preprint arXiv:1807.04836, 2018.\n\n[40] Xianjun Xia, Roberto Togneri, Ferdous Sohel, and David Huang. Auxiliary classi\ufb01er generative adversarial\nnetwork with soft labels in imbalanced acoustic event detection. IEEE Transactions on Multimedia, 2018.\n\n[41] Hang Zhou, Yu Liu, Ziwei Liu, Ping Luo, and Xiaogang Wang. Talking face generation by adversarially\n\ndisentangled audio-visual representation. arXiv preprint arXiv:1807.07860, 2018.\n\n10\n\n\f", "award": [], "sourceid": 2846, "authors": [{"given_name": "Yandong", "family_name": "Wen", "institution": "Carnegie Mellon University"}, {"given_name": "Bhiksha", "family_name": "Raj", "institution": "Carnegie Mellon University"}, {"given_name": "Rita", "family_name": "Singh", "institution": "Carnegie Mellon University"}]}