{"title": "Universal Style Transfer via Feature Transforms", "book": "Advances in Neural Information Processing Systems", "page_first": 386, "page_last": 396, "abstract": "Universal style transfer aims to transfer arbitrary visual styles to content images. Existing feed-forward based methods, while enjoying the inference efficiency, are mainly limited by inability of generalizing to unseen styles or compromised visual quality. In this paper, we present a simple yet effective method that tackles these limitations without training on any pre-defined styles. The key ingredient of our method is a pair of feature transforms, whitening and coloring, that are embedded to an image reconstruction network. The whitening and coloring transforms reflect direct matching of feature covariance of the content image to a given style image, which shares similar spirits with the optimization of Gram matrix based cost in neural style transfer. We demonstrate the effectiveness of our algorithm by generating high-quality stylized images with comparisons to a number of recent methods. We also analyze our method by visualizing the whitened features and synthesizing textures by simple feature coloring.", "full_text": "Universal Style Transfer via Feature Transforms\n\nYijun Li\nUC Merced\n\nyli62@ucmerced.edu\n\nChen Fang\n\nAdobe Research\n\ncfang@adobe.com\n\nZhaowen Wang\nAdobe Research\n\nzhawang@adobe.com\n\nXin Lu\n\nAdobe Research\nxinl@adobe.com\n\nAbstract\n\nJimei Yang\n\nAdobe Research\n\njimyang@adobe.com\n\nMing-Hsuan Yang\n\nUC Merced, NVIDIA Research\n\nmhyang@ucmerced.edu\n\nUniversal style transfer aims to transfer arbitrary visual styles to content images.\nExisting feed-forward based methods, while enjoying the inference ef\ufb01ciency, are\nmainly limited by inability of generalizing to unseen styles or compromised visual\nquality. In this paper, we present a simple yet effective method that tackles these\nlimitations without training on any pre-de\ufb01ned styles. The key ingredient of our\nmethod is a pair of feature transforms, whitening and coloring, that are embedded\nto an image reconstruction network. The whitening and coloring transforms re\ufb02ect\na direct matching of feature covariance of the content image to a given style\nimage, which shares similar spirits with the optimization of Gram matrix based\ncost in neural style transfer. We demonstrate the effectiveness of our algorithm by\ngenerating high-quality stylized images with comparisons to a number of recent\nmethods. We also analyze our method by visualizing the whitened features and\nsynthesizing textures via simple feature coloring.\n\n1\n\nIntroduction\n\nStyle transfer is an important image editing task which enables the creation of new artistic works.\nGiven a pair of examples, i.e., the content and style image, it aims to synthesize an image that\npreserves some notion of the content but carries characteristics of the style. The key challenge is how\nto extract effective representations of the style and then match it in the content image. The seminal\nwork by Gatys et al. [8, 9] show that the correlation between features, i.e., Gram matrix or covariance\nmatrix (shown to be as effective as Gram matrix in [20]), extracted by a trained deep neural network\nhas remarkable ability of capturing visual styles. Since then, signi\ufb01cant efforts have been made to\nsynthesize stylized images by minimizing Gram/covariance matrices based loss functions, through\neither iterative optimization [9] or trained feed-forward networks [27, 16, 20, 2, 6]. Despite the recent\nrapid progress, these existing works often trade off between generalization, quality and ef\ufb01ciency,\nwhich means that optimization-based methods can handle arbitrary styles with pleasing visual quality\nbut at the expense of high computational costs, while feed-forward approaches can be executed\nef\ufb01ciently but are limited to a \ufb01xed number of styles or compromised visual quality.\nBy far, the problem of universal style transfer remains a daunting task as it is challenging to develop\nneural networks that achieve generalization, quality and ef\ufb01ciency at the same time. The main issue\nis how to properly and effectively apply the extracted style characteristics (feature correlations) to\ncontent images in a style-agnostic manner.\nIn this work, we propose a simple yet effective method for universal style transfer, which enjoys\nthe style-agnostic generalization ability with marginally compromised visual quality and execution\nef\ufb01ciency. The transfer task is formulated as image reconstruction processes, with the content features\n\n31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA.\n\n\f(c) Multi-level stylization\n\n(a) Reconstruction (b) Single-level stylization\nFigure 1: Universal style transfer pipeline. (a) We \ufb01rst pre-train \ufb01ve decoder networks DecoderX\n(X=1,2,...,5) through image reconstruction to invert different levels of VGG features. (b) With both\nVGG and DecoderX \ufb01xed, and given the content image C and style image S, our method performs\nthe style transfer through whitening and coloring transforms. (c) We extend single-level to multi-level\nstylization in order to match the statistics of the style at all levels. The result obtained by matching\nhigher level statistics of the style is treated as the new content to continue to match lower-level\ninformation of the style.\n\nbeing transformed at intermediate layers with regard to the statistics of the style features, in the\nmidst of feed-forward passes. In each intermediate layer, our main goal is to transform the extracted\ncontent features such that they exhibit the same statistical characteristics as the style features of the\nsame layer and we found that the classic signal whitening and coloring transforms (WCTs) on those\nfeatures are able to achieve this goal in an almost effortless manner.\nIn this work, we \ufb01rst employ the VGG-19 network [26] as the feature extractor (encoder), and train a\nsymmetric decoder to invert the VGG-19 features to the original image, which is essentially the image\nreconstruction task (Figure 1(a)). Once trained, both the encoder and the decoder are \ufb01xed through all\nthe experiments. To perform style transfer, we apply WCT to one layer of content features such that\nits covariance matrix matches that of style features, as shown in Figure 1(b). The transformed features\nare then fed forward into the downstream decoder layers to obtain the stylized image. In addition\nto this single-level stylization, we further develop a multi-level stylization pipeline, as depicted in\nFigure 1(c), where we apply WCT sequentially to multiple feature layers. The multi-level algorithm\ngenerates stylized images with greater visual quality, which are comparable or even better with much\nless computational costs. We also introduce a control parameter that de\ufb01nes the degree of style\ntransfer so that the users can choose the balance between stylization and content preservation. The\nentire procedure of our algorithm only requires learning the image reconstruction decoder with no\nstyle images involved. So when given a new style, we simply need to extract its feature covariance\nmatrices and apply them to the content features via WCT. Note that this learning-free scheme is\nfundamentally different from existing feed-forward networks that require learning with pre-de\ufb01ned\nstyles and \ufb01ne-tuning for new styles. Therefore, our approach is able to achieve style transfer\nuniversally.\nThe main contributions of this work are summarized as follows:\n\nfeature statistics to those of a style image in the deep feature space.\n\n\u2022 We propose to use feature transforms, i.e., whitening and coloring, to directly match content\n\u2022 We couple the feature transforms with a pre-trained general encoder-decoder network, such\n\u2022 We demonstrate the effectiveness of our method for universal style transfer with high-quality\n\nthat the transferring process can be implemented by simple feed-forward operations.\n\nvisual results, and also show its application to universal texture synthesis.\n\n2 Related Work\n\nExisting style transfer methods are mostly example-based [13, 25, 24, 7, 21]. The image analogy\nmethod [13] aims to determine the relationship between a pair of images and then apply it to stylize\n\n2\n\nVGG Relu_X_1ReconsDecoderXVGG Relu_X_1ReconsDecoderXWhitening & Coloring Transform (WCT)OutputCSVGGRelu_5_1 ReconsDecoder5WCTVGGRelu_4_1 ReconsDecoder4VGGRelu_3_1 ReconsDecoder3VGGRelu_2_1 ReconsDecoder2VGGRelu_1_1 ReconsDecoder1WCTWCTWCTWCTOutputCSSSSSI5I4I3I2I1\fother images. As it is based on \ufb01nding dense correspondence, analogy-based approaches [25, 24, 7,\n21] often require that a pair of image depicts the same type of scene. Therefore these methods do not\nscale to the setting of arbitrary style images well.\nRecently, Gatys et al. [8, 9] proposed an algorithm for arbitrary stylization based on matching the\ncorrelations (Gram matrix) between deep features extracted by a trained network classi\ufb01er within an\niterative optimization framework. Numerous methods have since been developed to address different\naspects including speed [27, 19, 16], quality [28, 18, 32, 31], user control [10], diversity [29, 20],\nsemantics understanding [7, 1] and photorealism [23].\nIt is worth mentioning that one of the\nmajor drawbacks of [8, 9] is the inef\ufb01ciency due to the optimization process. The improvement of\nef\ufb01ciency in [27, 19, 16] is realized by formulating the stylization as learning a feed-forward image\ntransformation network. However, these methods are limited by the requirement of training one\nnetwork per style due to the lack of generalization in network design.\nMost recently, a number of methods have been proposed to empower a single network to transfer\nmultiple styles, including a model that conditioned on binary selection units [20], a network that\nlearns a set of new \ufb01lters for every new style [2], and a novel conditional normalization layer that\nlearns normalization parameters for each style [6]. To achieve arbitrary style transfer, Chen et al. [3]\n\ufb01rst propose to swap the content feature with the closest style feature locally. Meanwhile, inspired\nby [6], two following work [30, 11] turn to learn a general mapping from the style image to style\nparameters. One closest related work [15] directly adjusts the content feature to match the mean and\nvariance of the style feature. However, the generalization ability of the learned models on unseen\nstyles is still limited.\nDifferent from the existing methods, our approach performs style transfer ef\ufb01ciently in a feed-forward\nmanner while achieving generalization and visual quality on arbitrary styles. Our approach is closely\nrelated to [15], where content feature in a particular (higher) layer is adaptively instance normalized\nby the mean and variance of style feature. This step can be viewed as a sub-optimal approximation\nof the WCT operation, thereby leading to less effective results on both training and unseen styles.\nMoreover, our encoder-decoder network is trained solely based on image reconstruction, while [15]\nrequires learning such a module particularly for stylization task. We evaluate the proposed algorithm\nwith existing approaches extensively on both style transfer and texture synthesis tasks and present\nin-depth analysis.\n\n3 Proposed Algorithm\n\nWe formulate style transfer as an image reconstruction process coupled with feature transformation,\ni.e., whitening and coloring. The reconstruction part is responsible for inverting features back to the\nRGB space and the feature transformation matches the statistics of a content image to a style image.\n\n3.1 Reconstruction decoder\n\nWe construct an auto-encoder network for general image reconstruction. We employ the VGG-19 [26]\nas the encoder, \ufb01x it and train a decoder network simply for inverting VGG features to the original\nimage, as shown in Figure 1(a). The decoder is designed as being symmetrical to that of VGG-19\nnetwork (up to Relu_X_1 layer), with the nearest neighbor upsampling layer used for enlarging\nfeature maps. To evaluate with features extracted at different layers, we select feature maps at \ufb01ve\nlayers of the VGG-19, i.e., Relu_X_1 (X=1,2,3,4,5), and train \ufb01ve decoders accordingly. The pixel\nreconstruction loss [5] and feature loss [16, 5] are employed for reconstructing an input image,\n\nL = (cid:107)Io \u2212 Ii(cid:107)2\n\n2 + \u03bb(cid:107)\u03a6(Io) \u2212 \u03a6(Ii)(cid:107)2\n2 ,\n\n(1)\nwhere Ii, Io are the input image and reconstruction output, and \u03a6 is the VGG encoder that extracts\nthe Relu_X_1 features. In addition, \u03bb is the weight to balance the two losses. After training, the\ndecoder is \ufb01xed (i.e., will not be \ufb01ne-tuned) and used as a feature inverter.\n\n3.2 Whitening and coloring transforms\n\nGiven a pair of content image Ic and style image Is, we \ufb01rst extract their vectorized VGG feature\nmaps fc \u2208 (cid:60)C\u00d7HcWc and fs \u2208 (cid:60)C\u00d7HsWs at a certain layer (e.g., Relu_4_1), where Hc, Wc (Hs,\n\n3\n\n\fFigure 2: Inverting whitened features. We invert the whitened VGG Relu_4_1 feature as an example.\nLeft: original images, Right: inverted results (pixel intensities are rescaled for better visualization).\nThe whitened features still maintain global content structures.\n\n(a) Style\n\n(b) Content\n\n(c) HM (d) WCT\n\n(e) Style\n\n(f) Content\n\n(g) HM (h) WCT\n\nFigure 3: Comparisons between different feature transform strategies. Results are obtained by our\nmulti-level stylization framework in order to match all levels of information of the style.\n\nWs) are the height and width of the content (style) feature, and C is the number of channels. The\ndecoder will reconstruct the original image Ic if fc is directly fed into it. We next propose to use a\nwhitening and coloring transform to adjust fc with respect to the statistics of fs. The goal of WCT\nis to directly transform the fc to match the covariance matrix of fs. It consists of two steps, i.e.,\nwhitening and coloring transform.\n\nWhitening transform. Before whitening, we \ufb01rst center fc by subtracting its mean vector mc.\nThen we transform fc linearly as in (2) so that we obtain \u02c6fc such that the feature maps are uncorrelated\n( \u02c6fc\n\n= I),\n\n\u02c6fc\n\n(cid:62)\n\n\u02c6fc = Ec D\n\nc fc ,\n\n\u2212 1\nc E(cid:62)\n\n2\n\n(2)\nc \u2208 (cid:60)C\u00d7C, and Ec\n\nwhere Dc is a diagonal matrix with the eigenvalues of the covariance matrix fc f(cid:62)\nis the corresponding orthogonal matrix of eigenvectors, satisfying fc f(cid:62)\nTo validate what is encoded in the whitened feature \u02c6fc, we invert it to the RGB space with our\nprevious decoder trained for reconstruction only. Figure 2 shows two visualization examples, which\nindicate that the whitened features still maintain global structures of the image contents, but greatly\nhelp remove other information related to styles. We note especially that, for the Starry_night example\non right, the detailed stroke patterns across the original image are gone. In other words, the whitening\nstep helps peel off the style from an input image while preserving the global content structure. The\noutcome of this operation is ready to be transformed with the target style.\n\nc = EcDcE(cid:62)\nc .\n\nColoring transform. We \ufb01rst center fs by subtracting its mean vector ms, and then carry out\nthe coloring transform [14], which is essentially the inverse of the whitening step to transform \u02c6fc\nlinearly as in (3) such that we obtain \u02c6fcs which has the desired correlations between its feature maps\n( \u02c6fcs\n\n= fs f(cid:62)\ns ),\n\n\u02c6fcs\n\n(cid:62)\n\ns\n\n\u02c6fc ,\n\n\u02c6fcs = Es D\n\n(3)\ns \u2208 (cid:60)C\u00d7C, and Es\nwhere Ds is a diagonal matrix with the eigenvalues of the covariance matrix fs f(cid:62)\nis the corresponding orthogonal matrix of eigenvectors. Finally we re-center the \u02c6fcs with the mean\nvector ms of the style, i.e., \u02c6fcs = \u02c6fcs + ms.\nTo demonstrate the effectiveness of WCT, we compare it with a commonly used feature adjustment\ntechnique, i.e., histogram matching (HM), in Figure 3. The channel-wise histogram matching [12]\nmethod determines a mapping function such that the mapped fc has the same cumulative histogram as\nfs. In Figure 3, it is clear that the HM method helps transfer the global color of the style image well\n\n1\n2\n\ns E(cid:62)\n\n4\n\n\f(a) Style\n\n(b) Relu_1_1\n\n(c) Relu_2_1\n\n(d) Relu_3_1\n\n(e) Relu_4_1\n\n(f) Relu_5_1\n\nFigure 4: Single-level stylization using different VGG features. The content image is from Figure 2.\n\n(a) I5\n\n(b) I4\n\n(c) I1\n\n(d) Fine-to-coarse\n\nFigure 5: (a)-(c) Intermediate results of our coarse-to-\ufb01ne multi-level stylization framework in\nFigure 1(c). The style and content images are from Figure 4. I1 is the \ufb01nal output of our multi-level\npipeline. (d) Reversed \ufb01ne-to-coarse multi-level pipeline.\n\nbut fails to capture salient visual patterns, e.g., patterns are broken into pieces and local structures are\nmisrepresented. In contrast, our WCT captures patterns that re\ufb02ect the style image better. This can\nbe explained by that the HM method does not consider the correlations between features channels,\nwhich are exactly what the covariance matrix is designed for.\nAfter the WCT, we may blend \u02c6fcs with the content feature fc as in (4) before feeding it to the decoder\nin order to provide user controls on the strength of stylization effects:\n\n\u02c6fcs = \u03b1 \u02c6fcs + (1 \u2212 \u03b1) fc ,\n\n(4)\n\nwhere \u03b1 serves as the style weight for users to control the transfer effect.\n\n3.3 Multi-level coarse-to-\ufb01ne stylization\n\nBased on the single-level stylization framework shown in Figure 1(b), we use different layers of VGG\nfeatures Relu_X_1 (X=1,2,...,5) and show the corresponding stylized results in Figure 4. It clearly\nshows that the higher layer features capture more complicated local structures, while lower layer\nfeatures carry more low-level information (e.g., colors). This can be explained by the increasing size\nof receptive \ufb01eld and feature complexity in the network hierarchy. Therefore, it is advantageous to\nuse features at all \ufb01ve layers to fully capture the characteristics of a style from low to high levels.\nFigure 1(c) shows our multi-level stylization pipeline. We start by applying the WCT on Relu_5_1\nfeatures to obtain a coarse stylized result and regard it as the new content image to further adjust\nfeatures in lower layers. An example of intermediate results are shown in Figure 5. We show the\nintermediate results I5, I4, I1 with obvious differences, which indicates that the higher layer features\n\ufb01rst capture salient patterns of the style and lower layer features further improve details. If we reverse\nfeature processing order (i.e., \ufb01ne-to-coarse layers) by starting with Relu_1_1, low-level information\ncannot be preserved after manipulating higher level features, as shown in Figure 5(d).\n\n4 Experimental Results\n\n4.1 Decoder training\n\nFor the multi-level stylization approach, we separately train \ufb01ve reconstruction decoders for features\nat the VGG-19 Relu_X_1 (X=1,2,...,5) layer. It is trained on the Microsoft COCO dataset [22] and\nthe weight \u03bb to balance the two losses in (1) is set as 1.\n\n5\n\n\f(a) Style\n\n(b) [3]\n\n(c) [15]\n\n(d) [27]\n\n(e) [9]\n\n(f) Ours\n\nFigure 6: Results from different style transfer methods. The content images are from Figure 2-3. We\nevaluate various styles including paintings, abstract styles, and styles with obvious texton elements.\nWe adjust the style weight of each method to obtain the best stylized effect. For our results, we set\nthe style weight \u03b1 = 0.6.\n\nTable 1: Differences between our approach and other methods.\n\nChen et al. [3] Huang et al. [15] TNet [27] DeepArt [9] Ours\n\u221a\n\u221a\n\u221a\n\n\u221a\n\u00d7\n\u221a\n\n\u221a\n\u221a\n\u00d7\n\n\u221a\n\u221a\n\u00d7\n\n\u00d7\n\u221a\n\u00d7\n\nArbitrary\nEf\ufb01cient\nLearning-free\n\n4.2 Style transfer\n\nTo demonstrate the effectiveness of the proposed algorithm, we list the differences with existing\nmethods in Table 1 and present stylized results in Figure 6. We adjust the style weight of other\nmethods to obtain the best stylized effect. The optimization-based work of [9] handles arbitrary\nstyles but is likely to encounter unexpected local minima issues (e.g., 5th and 6th row of Figure 6(e)).\nAlthough the method [27] greatly improves the stylization speed, it trades off quality and generality\nfor ef\ufb01ciency, which generates repetitive patterns that overlay with the image contents (Figure 6(d)).\n\n6\n\n\fTable 2: Quantitative comparisons between different stylization methods in terms of the covariance\nmatrix difference (Ls), user preference and run-time, tested on images of size 256 \u00d7 256 and a 12GB\nTITAN X.\n\nlog(Ls)\nPreference/%\nTime/sec\n\nChen et al. [3] Huang et al. [15] TNet [27] Gatys et al. [9] Ours\n6.3\n30.3\n0.83\n\n7.4\n15.7\n2.1\n\n7.0\n24.9\n0.20\n\n6.8\n12.7\n0.18\n\n6.7\n16.4\n21.2\n\nStyle\n\nscale = 256\n\nscale = 768\n\n\u03b1 = 0.4\n\n\u03b1 = 0.6\n\n\u03b1 = 1.0\n\nFigure 7: Controlling the stylization on the scale and weight.\n\nClosest to our work on generalization are the recent methods [3, 15], but the quality of the stylized\nresults are less appealing. The work of [3] replaces the content feature with the most similar style\nfeature based on patch similarity and hence has limited capability, i.e., the content is strictly preserved\nwhile style is not well re\ufb02ected with only low-level information (e.g., colors) transferred, as shown\nin Figure 6(b). In [15], the content feature is simply adjusted to have the same mean and variance\nwith the style feature, which is not effective in capturing high-level representations of the style. Even\nlearned with a set of training styles, it does not generalize well on unseen styles. Results in Figure 6(c)\nindicate that the method in [15] is not effective at capturing and synthesizing salient style patterns,\nespecially for complicated styles where there are rich local structures and non-smooth regions.\nFigure 6(f) shows the stylized results of our approach. Without learning any style, our method is able\nto capture visually salient patterns in style images (e.g., the brick wall on the 6th row). Moreover,\nkey components in the content images (e.g., bridge, eye, mouth) are also well stylized in our results,\nwhile other methods only transfer patterns to relatively smooth regions (e.g., sky, face). The models\nand code are available at https://github.com/Yijunmaverick/UniversalStyleTransfer.\nIn addition, we quantitatively evaluate different methods by computing the covariance matrix differ-\nence (Ls) on all \ufb01ve levels of VGG features between stylized results and the given style image. We\nrandomly select 10 content images from [22] and 40 style images from [17], compute the averaged\ndifference over all styles, and show the results in Table 2 (1st row). Quantitative results show that we\ngenerate stylized results with lower Ls, i.e., closer to the statistics of the style.\n\nUser study. Evaluating artistic style transfer has been an open question in the community. Since\nthe qualitative assessment is highly subjective, we conduct a user study to evaluate 5 methods shown\nin Figure 6. We use 5 content images and 30 style images, and generate 150 results based on each\ncontent/style pair for each method. We randomly select 15 style images for each subject to evaluate.\nWe display stylized images by 5 compared methods side-by-side on a webpage in random order. Each\nsubject is asked to vote his/her ONE favorite result for each style. We \ufb01nally collect the feedback\nfrom 80 subjects of totally 1,200 votes and show the percentage of the votes each method received in\nTable 2 (2nd row). The study shows that our method receives the most votes for better stylized results.\nIt can be an interesting direction to develop evaluation metrics based on human visual perception for\ngeneral image synthesis problems.\n\nEf\ufb01ciency.\nIn Table 2 (3rd row), we also compare our approach with other methods in terms of\nef\ufb01ciency. The method by Gatys et al. [9] is slow due to loops of optimization and usually requires at\nleast 500 iterations to generate good results. The methods [27] and [15] are ef\ufb01cient as the scheme is\nbased on one feed-forward pass with a trained network. The approach [3] is feed-forward based but\nrelatively slower as the feature swapping operation needs to be carried out for thousands of patches.\nOur approach is also ef\ufb01cient but a little bit slower than [27, 15] because we have a eigenvalue\ndecomposition step in WCT. But note that the computational cost on this step will not increase along\nwith the image size because the the dimension of covariance matrix only depends on \ufb01lter numbers (or\n\n7\n\n\f(a) Content\n\n(b) Different masks and styles\n\n(c) Our result\n\nFigure 8: Spatial control in transferring, which enables users to edit the content with different styles.\n\nFigure 9: Texture synthesis. In each panel, Left: original textures, Right: our synthesized results.\nTexture images are mostly from the Describable Textures Dataset (DTD) [4].\n\nchannels), which is at most 512 (Relu_5_1). Currently the decomposition step is implemented based\non CPU. Our future work includes more ef\ufb01cient GPU implementations of the proposed algorithm.\n\nUser Controls. Given a content/style pair, our approach is not only as simple as a one-click\ntransferring, but also \ufb02exible enough to accommodate different requirements from users by providing\ndifferent controls on the stylization, including the scale, weight and spatial control. The style input\non different scales will lead to different extracted statistics due to the \ufb01xed receptive \ufb01eld of the\nnetwork. Therefore the scale control is easily achieved by adjusting the style image size. In the\nmiddle of Figure 7, we show two examples where the brick can be transferred in either small or\nlarge scale. The weight control refers to controlling the balance between stylization and content\npreservation. As shown on right of Figure 7, our method enjoys this \ufb02exibility in simple feed-forward\npasses by simply adjusting the style weight \u03b1 in (4). However in [9] and [27], to obtain visual results\nof different weight settings, a new round of time-consuming optimization or model training is needed.\nMoreover, our blending directly works on deep feature space before inversion/reconstruction, which\nis fundamentally different from [9, 27] where the blending is formulated as the weighted sum of the\ncontent and style losses that may not always lead to a good balance point.\nThe spatial control is also highly desired when users want to edit an image with different styles\ntransferred on different parts of the image. Figure 8 shows an example of spatially controlling the\nstylization. A set of masks M (Figure 8(b)) is additionally required as input to indicate the spatial\ncorrespondence between content regions and styles. By replacing the content feature fc in (3) with\nM (cid:12) fc where (cid:12) is a simple mask-out operation, we are able to stylize the speci\ufb01ed region only.\n\n4.3 Texture synthesis\n\nBy setting the content image as a random noise image (e.g., Gaussian noise), our stylization framework\ncan be easily applied to texture synthesis. An alternative is to directly initialize the \u02c6fc in (3) to be white\nnoise. Both approaches achieve similar results. Figure 9 shows a few examples of the synthesized\ntextures. We empirically \ufb01nd that it is better to run the multi-level pipeline for a few times (e.g., 3) to\nget more visually pleasing results.\nOur method is also able to synthesize the interpolated result of two textures. Given two texture\nexamples s1 and s2, we \ufb01rst perform the WCT on the input noise and get transformed features \u02c6fcs1\nand \u02c6fcs2 respectively. Then we blend these two features \u02c6fcs = \u03b2 \u02c6fcs1 + (1 \u2212 \u03b2) \u02c6fcs2 and feed the\n\n8\n\n\fTexture s1\n\nTexture s2\n\n\u03b2 = 0.75\n\n\u03b2 = 0.5\n\n\u03b2 = 0.25\n\n\u03b2 = 0.5\n\nFigure 10: Interpolation between two texture examples. Left: original textures, Middle: our interpola-\ntion results, Right: interpolated results of [9]. \u03b2 controls the weight of interpolation.\n\nTexture\n\nTNet [27]\n\nOurs\n\nFigure 11: Comparisons of diverse synthesized results between TNet [27] and our model.\n\ncombined feature into the decoder to generate mixed effects. Note that our interpolation directly\nworks on deep feature space. By contrast, the method in [9] generates the interpolation by matching\nthe weighted sum of Gram matrices of two textures at the loss end. Figure 10 shows that the result\nby [9] is simply overlaid by two textures while our method generates new textural effects, e.g., bricks\nin the stripe shape.\nOne important aspect in texture synthesis is diversity. By sampling different noise images, our\nmethod can generate diverse synthesized results for each texture. While [27] can generate different\nresults driven by the input noise, the learned networks are very likely to be trapped in local optima. In\nother words, the noise is marginalized out and thus fails to drive the network to generate large visual\nvariations. In contrast, our approach explains each input noise better because the network is unlikely\nto absorb the variations in input noise since it is never trained for learning textures. We compare the\ndiverse outputs of our model with [27] in Figure 11. Note that the common diagonal layout is shared\nacross different results of [27], which causes unsatisfying visual experiences. The comparison shows\nthat our method achieves diversity in a more natural and \ufb02exible manner.\n\n5 Concluding Remarks\n\nIn this work, we propose a universal style transfer algorithm that does not require learning for each\nindividual style. By unfolding the image generation process via training an auto-encoder for image\nreconstruction, we integrate the whitening and coloring transforms in the feed-forward passes to\nmatch the statistical distributions and correlations between the intermediate features of content and\nstyle. We also present a multi-level stylization pipeline, which takes all level of information of a\nstyle into account, for improved results. In addition, the proposed approach is shown to be equally\neffective for texture synthesis. Experimental results demonstrate that the proposed algorithm achieves\nfavorable performance against the state-of-the-art methods in generalizing to arbitrary styles.\n\nAcknowledgments\n\nThis work is supported in part by the NSF CAREER Grant #1149783, gifts from Adobe and NVIDIA.\n\n9\n\n\fReferences\n[1] A. J. Champandard. Semantic style transfer and turning two-bit doodles into \ufb01ne artworks. arXiv preprint\n\narXiv:1603.01768, 2016.\n\n[2] D. Chen, L. Yuan, J. Liao, N. Yu, and G. Hua. Stylebank: An explicit representation for neural image style\n\ntransfer. In CVPR, 2017.\n\n[3] T. Q. Chen and M. Schmidt. Fast patch-based style transfer of arbitrary style.\n\narXiv:1612.04337, 2016.\n\narXiv preprint\n\n[4] M. Cimpoi, S. Maji, I. Kokkinos, S. Mohamed, , and A. Vedaldi. Describing textures in the wild. In CVPR,\n\n2014.\n\n[5] A. Dosovitskiy and T. Brox. Generating images with perceptual similarity metrics based on deep networks.\n\nIn NIPS, 2016.\n\n[6] V. Dumoulin, J. Shlens, and M. Kudlur. A learned representation for artistic style. In ICLR, 2017.\n\n[7] O. Frigo, N. Sabater, J. Delon, and P. Hellier. Split and match: Example-based adaptive patch sampling for\n\nunsupervised style transfer. In CVPR, 2016.\n\n[8] L. A. Gatys, A. S. Ecker, and M. Bethge. Texture synthesis using convolutional neural networks. In NIPS,\n\n2015.\n\n[9] L. A. Gatys, A. S. Ecker, and M. Bethge. Image style transfer using convolutional neural networks. In\n\nCVPR, 2016.\n\n[10] L. A. Gatys, A. S. Ecker, M. Bethge, A. Hertzmann, and E. Shechtman. Controlling perceptual factors in\n\nneural style transfer. In CVPR, 2017.\n\n[11] G. Ghiasi, H. Lee, M. Kudlur, V. Dumoulin, and J. Shlens. Exploring the structure of a real-time, arbitrary\n\nneural artistic stylization network. In BMVC, 2017.\n\n[12] R. C. Gonzalez and R. E. Woods. Digital image processing (3rd edition). Prentice Hall, 2008.\n\n[13] A. Hertzmann, C. E. Jacobs, N. Oliver, B. Curless, and D. H. Salesin. Image analogies. In SIGGRAPH,\n\n2001.\n\n[14] M. Hossain. Whitening and coloring transforms for multivariate gaussian random variables. Project Rhea,\n\n2016.\n\n[15] X. Huang and S. Belongie. Arbitrary style transfer in real-time with adaptive instance normalization. In\n\nICCV, 2017.\n\n[16] J. Johnson, A. Alahi, and L. Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In\n\nECCV, 2016.\n\n[17] S. Karayev, M. Trentacoste, H. Han, A. Agarwala, T. Darrell, A. Hertzmann, and H. Winnemoeller.\n\nRecognizing image style. In BMVC, 2014.\n\n[18] C. Li and M. Wand. Combining markov random \ufb01elds and convolutional neural networks for image\n\nsynthesis. In CVPR, 2016.\n\n[19] C. Li and M. Wand. Precomputed real-time texture synthesis with markovian generative adversarial\n\nnetworks. In ECCV, 2016.\n\n[20] Y. Li, C. Fang, J. Yang, Z. Wang, X. Lu, and M.-H. Yang. Diversi\ufb01ed texture synthesis with feed-forward\n\nnetworks. In CVPR, 2017.\n\n[21] J. Liao, Y. Yao, L. Yuan, G. Hua, and S. B. Kang. Visual attribute transfer through deep image analogy.\n\narXiv preprint arXiv:1705.01088, 2017.\n\n[22] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Doll\u00e1r, and C. L. Zitnick. Microsoft\n\nCOCO: Common objects in context. In ECCV, 2014.\n\n[23] F. Luan, S. Paris, E. Shechtman, and K. Bala. Deep photo style transfer. In CVPR, 2017.\n\n[24] Y. Shih, S. Paris, C. Barnes, W. T. Freeman, and F. Durand. Style transfer for headshot portraits. In\n\nSIGGRAPH, 2014.\n\n10\n\n\f[25] Y. Shih, S. Paris, F. Durand, and W. T. Freeman. Data-driven hallucination of different times of day from a\n\nsingle outdoor photo. In SIGGRAPH, 2013.\n\n[26] K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition.\n\narXiv preprint arXiv:1409.1556, 2014.\n\n[27] D. Ulyanov, V. Lebedev, A. Vedaldi, and V. Lempitsky. Texture networks: Feed-forward synthesis of\n\ntextures and stylized images. In ICML, 2016.\n\n[28] D. Ulyanov, A. Vedaldi, and V. Lempitsky.\n\nInstance normalization: The missing ingredient for fast\n\nstylization. arXiv preprint arXiv:1607.08022, 2016.\n\n[29] D. Ulyanov, A. Vedaldi, and V. Lempitsky. Improved texture networks: Maximizing quality and diversity\n\nin feed-forward stylization and texture synthesis. In CVPR, 2017.\n\n[30] H. Wang, X. Liang, H. Zhang, D.-Y. Yeung, and E. P. Xing. Zm-net: Real-time zero-shot image manipula-\n\ntion network. arXiv preprint arXiv:1703.07255, 2017.\n\n[31] X. Wang, G. Oxholm, D. Zhang, and Y.-F. Wang. Multimodal transfer: A hierarchical deep convolutional\n\nneural network for fast artistic style transfer. In CVPR, 2017.\n\n[32] P. Wilmot, E. Risser, and C. Barnes. Stable and controllable neural texture synthesis and style transfer\n\nusing histogram losses. arXiv preprint arXiv:1701.08893, 2017.\n\n11\n\n\f", "award": [], "sourceid": 295, "authors": [{"given_name": "Yijun", "family_name": "Li", "institution": "University of California, Merced"}, {"given_name": "Chen", "family_name": "Fang", "institution": "Adobe Research"}, {"given_name": "Jimei", "family_name": "Yang", "institution": "Adobe Research"}, {"given_name": "Zhaowen", "family_name": "Wang", "institution": "Adobe Research"}, {"given_name": "Xin", "family_name": "Lu", "institution": "Adobe"}, {"given_name": "Ming-Hsuan", "family_name": "Yang", "institution": "UC Merced"}]}