{"title": "Learning Sparse Multiscale Image Representations", "book": "Advances in Neural Information Processing Systems", "page_first": 1351, "page_last": 1358, "abstract": null, "full_text": "Learning Sparse Multiscale Image\n\nRepresentations\n\nPhil Sallee\n\nBruno A. Olshausen\n\nDepartment of Computer Science and\n\nDepartment of Psychology and\n\nCenter for Neuroscience, UC Davis\n\nCenter for Neuroscience, UC Davis\n\n1544 Newton Ct.\nDavis, CA 95616\n\nsallee@cs.ucdavis.edu\n\n1544 Newton Ct.\nDavis, CA 95616\n\nbaolshausen@ucdavis.edu\n\nAbstract\n\nWe describe a method for learning sparse multiscale image repre-\nsentations using a sparse prior distribution over the basis function\ncoe(cid:14)cients. The prior consists of a mixture of a Gaussian and a\nDirac delta function, and thus encourages coe(cid:14)cients to have exact\nzero values. Coe(cid:14)cients for an image are computed by sampling\nfrom the resulting posterior distribution with a Gibbs sampler. The\nlearned basis is similar to the Steerable Pyramid basis, and yields\nslightly higher SNR for the same number of active coe(cid:14)cients. De-\nnoising using the learned image model is demonstrated for some\nstandard test images, with results that compare favorably with\nother denoising methods.\n\n1\n\nIntroduction\n\nIncreasing interest has been given to the use of overcomplete representations for\nnatural scenes, where the number of basis functions exceeds the number of image\npixels. One reason for this is that overcompleteness allows for more stable, and thus\narguably more meaningful, representations in which common image features can be\nwell described by only a few coe(cid:14)cients, regardless of where they are located in\nthe image, how they are rotated, or how large they are [8, 6]. This may translate\ninto gains in coding e(cid:14)ciency for image compression, and improved accuracy for\ntasks such as denoising. Overcomplete representations have been shown to reduce\nGibbs-like artifacts common to thresholding methods employing critically sampled\nwavelets [4, 3, 9].\n\nCommon wavelet denoising approaches generally apply either a hard or soft-\nthresholding function to coe(cid:14)cients which have been obtained by (cid:12)ltering an image\nwith a the basis functions. One can view these thresholding methods as a means\nof selecting coe(cid:14)cients for an image based on an assumed sparse prior on the co-\ne(cid:14)cients [1, 2]. This statistical framework provides a principled means of selecting\nan appropriate thresholding function. When such thresholding methods are applied\nto overcomplete representations, however, problems arise due to the dependencies\nbetween coe(cid:14)cients. Choosing optimal thresholds for a non-orthogonal basis is still\n\n\fan unsolved problem. In one approach, orthogonal subgroups of an overcomplete\nshift-invariant expansion are thresholded separately and then the results are com-\nbined by averaging [4, 3]. In addition, if the coe(cid:14)cients are obtained by (cid:12)ltering\nthe noisy image, there will be correlations in the noise that should be taken into\naccount.\n\nHere we address two major issues regarding the use of overcomplete representations\nfor images. First, current methods make use of various overcomplete wavelet bases.\nWhat is the optimal basis to use for a speci(cid:12)c class of data? To help answer this\nquestion, we describe how to adapt an overcomplete wavelet basis to the statis-\ntics of natural images. Secondly, we address the problem of properly inferring the\ncoe(cid:14)cients for an image when the basis is overcomplete. We avoid problems asso-\nciated with thresholding by using the wavelet basis as part of a generative model,\nrather than a simple (cid:12)ltering mechanism. We then sample the coe(cid:14)cients from\nthe resulting posterior distribution by simulating a Markov process known as a\nGibbs-sampler.\n\nOur previous work in this area made use of a prior distribution peaked at zero and\ntapering away smoothly to obtain sparse coe(cid:14)cients [7]. However, we encountered\na number of signi(cid:12)cant limitations with this method. First, the smooth priors\ndo not force inactive coe(cid:14)cients to have values exactly equal to zero, resulting in\ndecreased coding e(cid:14)ciency. E(cid:14)ciency may be partially regained by thresholding\nthe near-zero coe(cid:14)cients, but due to the non-orthogonality of the representation\nthis will produce sub-optimal results as previously mentioned. The maximum a\nposteriori (MAP) estimate also introduced biases in the learning process. These\ne(cid:11)ects can be partially compensated for by renormalizing the basis functions, but\nother parameters of the model such as those of the prior could not be learned.\nFinally, the gradient ascent method has convergence problems due to the power\nspectrum of natural images and the overcompleteness of the representation. Here we\nresolve these problems by using a prior distribution which is composed of a mixture\nof a Gaussian and a Dirac delta function, so that inactive coe(cid:14)cients are encouraged\nto have exact zero values. Similar models employing a mixture of two Gaussians\nhave been used for classifying wavelet coe(cid:14)cients into active (high variance) and\ninactive (low variance) states [2, 5]. Such a classi(cid:12)cation should be even more\nadvantageous if the basis is overcomplete. A method for performing Gibbs-sampling\nfor the Delta-plus-Gaussian prior in the context of an image pyramid is derived, and\ndemonstrated to be e(cid:11)ective at obtaining very sparse representations which match\nthe form of the imposed prior. Biases in the learning are overcome by sampling\ninstead of using a MAP estimate.\n\n2 Wavelet image model\n\nEach observed image I is assumed to be generated by a linear superposition of basis\nfunctions which are columns of an N by M weight matrix W, with the addition of\nGaussian noise (cid:23):\n\nI = W a + (cid:23);\n\n(1)\n\nwhere I is an N -element vector of image pixels and a is an M -element vector of basis\ncoe(cid:14)cients. In order to achieve a practical implementation which can be seamlessly\nscaled to any size image, we assume that the basis function matrix W is composed of\na small set of spatially localized mother wavelet functions  i(x; y), which are shifted\nto each position in the image and rescaled by factors of two. Unlike typical wavelet\ntransforms which use a single 1-D mother wavelet function to generate 2-D functions\nby inner product, we do not constrain the functions  i(x; y) to be 1-D separable.\n\n\fThe functions  i(x; y) provide an e(cid:14)cient way to perform computations involving\nW by means of convolutions. Basis functions of coarser scales are produced by\nupsampling the  i(x; y) functions and blurring with a low-pass (cid:12)lter (cid:30)(x; y), also\nknown as the scaling function. The image model above may be re-expressed to\nmake these parameters explicit:\n\n(2)\n\n(3)\n\nl < L (cid:0) 1\nl = L (cid:0) 1\n\nI(x; y) = g0(x; y) + (cid:23)(x; y)\n\ngl(x; y) = (cid:26) (cid:2)gl+1(x; y) \" 2(cid:3) (cid:3) (cid:30)(x; y) +Pi al\n\nal(x; y)\n\ni(x; y) (cid:3)  i(x; y)\n\nwhere the coe(cid:14)cients al\ni(x; y) are indexed by their position (x; y), band (i) and\nlevel of resolution (l) within the pyramid (l = 0 is the highest resolution level). The\nsymbol (cid:3) denotes convolution, and \" 2 denotes upsampling by two and is de(cid:12)ned as\n\nf (x; y) \" 2 (cid:17) (cid:26) f ( x\n\n0\n\n2 ; y\n\n2 ) x even & y even\n\notherwise\n\n(4)\n\nThe probability of generating an image I, given coe(cid:14)cients a, parameters (cid:18), assum-\ning Gaussian i.i.d. noise (cid:23) (with variance 1=(cid:21)N ), is\n\nP (Ija; (cid:18)) =\n\n1\nZ(cid:21)N\n\ne(cid:0)\n\n(cid:21)N\n2 jI(cid:0)W aj2\n\n:\n\n(5)\n\nThe prior probability over each coe(cid:14)cient ai is modeled as a mixture of a Gaussian\ndistribution and a Dirac delta function (cid:14)(ai). A binary state variable si for each\ncoe(cid:14)cient indicates whether the coe(cid:14)cient ai is active (any real value), or inactive\n(zero). The probability of a coe(cid:14)cient vector a given a binary state vector s and\nmodel parameters (cid:18) = fW; (cid:21)N ; (cid:21)a; (cid:3)sg is de(cid:12)ned as\n\nP (aijsi; (cid:18))\n\nP (ajs; (cid:18)) = Yi\nP (aijsi; (cid:18)) = ( (cid:14)(ai)\n\nZ(cid:21)ai\n\n1\n\n(cid:21)ai\n2 a2\n\ni\n\nif\n\nif\n\nsi = 0;\nsi = 1\n\ne(cid:0)\n\nwhere (cid:21)a is a vector with elements (cid:21)ai . The probability of a binary state s is\n\nP (sj(cid:18)) =\n\n1\n\nZ(cid:3)s\n\ne(cid:0) 1\n\n2\n\nsT (cid:3)s s:\n\n(6)\n\n(7)\n\n(8)\n\nMatrix (cid:3)s is assumed to be diagonal (for now), with nonzero elements (cid:21)si . The\nform of the prior is shown graphically in (cid:12)gure 1. Note that the parameters W, (cid:21)a,\nand (cid:3)s are themselves parameterized by a much smaller set of parameters. Only\nthe mother wavelet function  i(x; y) and a single (cid:21)si and (cid:21)ai parameter need to be\nlearned for each wavelet band, since we are assuming translation invariance.\n\nThe total image probability is obtained by marginalizing over the possible coe(cid:14)cient\nand state values:\n\nP (Ij(cid:18)) =Xs\n\nP (sj(cid:18))Z P (Ija; (cid:18))P (ajs; (cid:18)) da\n\n(9)\n\n3 Sampling and Inference\n\nWe show how to sample from the posterior distribution P (a; sjI; (cid:18)) for an image\nI using a Gibbs sampler. For each coe(cid:14)cient and state variable pair (ai,si), we\n\n\f10 0\n\n10 -1\n\n10 -2\n\n10 -3\n\n10 -4\n\n10 -5\n\n10 -6\n0.5\n\n0.4\n\n0.3\n\n0.2\n\n0.1\n\n0\n\n0.1\n\n0.2\n\n0.3\n\n0.4\n\n0.5\n\nFigure 1: Prior distribution (dashed), and histogram of samples taken from the\nposterior distribution (solid) plotted for a single coe(cid:14)cient. The y-axis is plotted\non a log scale.\n\nsample from the posterior distribution conditioned on the image and the remaining\ncoe(cid:14)cients a(cid:22)i: P (ai; sijI; a(cid:22)i; s(cid:22)i; (cid:18)). After all coe(cid:14)cients (and state variables) have\nbeen updated, this process is repeated until the system has reached equilibrium. To\ninfer an optimal representation a for an image I (for coding or denoising purposes),\nwe can either average a number of samples to estimate the posterior mean, or with\nminor adjustment locate a posterior maximum by raising the posterior distribution\nto a power (1=T ) and annealing T to zero. To sample from P (ai; sijI; a(cid:22)i; s(cid:22)i; (cid:18)), we\n(cid:12)rst draw a value for si from P (sijI; a(cid:22)i; s(cid:22)i; (cid:18)), then draw ai from P (aijsi; I; a(cid:22)i; s(cid:22)i; (cid:18)).\nFor P (sijI; a(cid:22)i; s(cid:22)i; (cid:18)) we have:\n\nwhere\n\nand\n\nP (sijI; a(cid:22)i; s(cid:22)i; (cid:18)) / P (sijs(cid:22)i; (cid:18))Z P (Ijai; a(cid:22)i; (cid:18))P (aijsi; (cid:18))dai\n\nP (sijs(cid:22)i; (cid:18)) =\n\nP (Ijai; a(cid:22)i; (cid:18)) =\n\n1\n\nZsijs(cid:22)i\n\ne(cid:0)\n\n(cid:21)si\n2 si;\n\n1\n\nZ(cid:21)ni\n\ne(cid:0)\n\n(cid:21)ni\n\n2 (ai(cid:0)bi)2\n\n;\n\n(cid:21)ni = (cid:21)N jWij2;\n\nbi =\n\nWi (cid:1) (I (cid:0) W ai=0)\n\njWij2\n\n:\n\n(10)\n\n(11)\n\n(12)\n\n(13)\n\nThe notation Wi denotes column i of matrix W, jWij is the length of vector Wi,\nand ai=0 denotes the current coe(cid:14)cient vector a except with ai set to zero. Thus,\nbi denotes the value for ai which minimizes the reconstruction error (while holding\na(cid:22)i constant). Since si can only take on two values, we can compute equation 10 for\nsi = 0 and si = 1, integrating over the possible coe(cid:14)cient values. This yields the\nfollowing sigmoidal activation rule as a function of bi:\n\nP (si = 1jI; a(cid:22)i; s(cid:22)i; (cid:18)) =\n\n1\n\n1 + e(cid:0)(cid:12)i(b2\n\ni (cid:0)ti)\n\nwhere\n\n(cid:12)i =\n\n1\n2\n\n(cid:21)2\nni\n\n(cid:21)ni + (cid:21)ai\n\n;\n\nti =\n\n(cid:21)ni + (cid:21)ai\n\n(cid:21)2\nni\n\n(cid:20)(cid:21)si (cid:0) log\n\n(cid:21)ai\n\n(cid:21)ni + (cid:21)ai(cid:21) :\n\n(14)\n\n(15)\n\n\fFor P (aijsi; I; a(cid:22)i; s(cid:22)i; (cid:18)) we have:\n\nP (aijsi; I; a(cid:22)i; s(cid:22)i; (cid:18)) = ( (cid:14)(ai)\n\nN ( (cid:21)ni bi\n(cid:21)ni +(cid:21)ai\n\n;\n\n1\n\n(cid:21)ni +(cid:21)ai\n\n)\n\nif\nif\n\nsi = 0;\nsi = 1\n\n(16)\n\nTo perform this procedure on a wavelet pyramid, the inner product computations\nnecessary to compute bi can be performed e(cid:14)ciently by means of convolutions with\nthe mother wavelet functions  i(x; y). The (cid:21)N ; (cid:21)si and (cid:21)ai parameters may be\nadapted to a speci(cid:12)c image during the inference process by use of the update rules\ndescribed in the next section. This method was found to be particularly useful for\ndenoising, when the variance of the noise was assumed to be unknown.\n\n4 Learning\n\nOur objective for learning is to adjust the parameters, (cid:18), to maximize the average\nlog-likelihood of images under the model:\n\n^(cid:18) = arg max\n\n(cid:18)\n\nhlog P (Ij(cid:18))i\n\n(17)\n\nThe parameters are updated by gradient ascent on this objective, which results in\nthe following update rules:\n\n(cid:1)(cid:21)si /\n\n1\n\n2*(cid:28)(cid:20)\n\n1\n\n1 + e\n\n1\n\n2 (cid:21)si\n\n(cid:0) si(cid:21)(cid:29)P (a;sjI;(cid:18))+\n\n(cid:1)(cid:21)ai /\n\n1\n\n2*(cid:28)si (cid:20) 1\n\n(cid:21)ai\n\n(cid:0) a2\n\ni(cid:21)(cid:29)P (a;sjI;(cid:18))+\n\n(18)\n\n(19)\n\n(20)\n\n(cid:1) i(x; y) / (cid:21)NDhe(x; y) ? ai(x; y)iP (a;sjI;(cid:18))E\n\nwhere ? denotes cross correlation and e(x; y) is the reconstruction error computed\nby e = I (cid:0) W a. Only a center portion of the cross correlation with the extent of\nthe  i(x; y) functions is computed to update the parameters. The outer brackets\ndenotes averaging over many images. The notation hiP () denotes averaging the\nquantity in brackets while sampling from the speci(cid:12)ed distribution.\n\n5 Results\n\nThe image model was trained on 22 512x512 pixel grayscale natural images (not\nwhitened). These images were generated from color images taken from a larger\ndatabase of photographic images 1. Smaller images (64x64 pixels) were selected\nrandomly for sampling during training. To simplify the learning procedure, sam-\npling was performed on a single spatial frequency scale. Each image was bandpass\n(cid:12)ltered for an octave range before sampling from the posterior for that scale. The\n\n1Images were downloaded from philip.greenspun.com with permission from Philip\n\nGreenspun.\n\n\f(a)\n\n(b)\n\nFigure 2: (a) Mother wavelet functions  i(x; y) adapted for 2, 4 and 6 bands and\ncorresponding power spectra showing power as a function of spatial frequency in\nthe 2D Fourier plane. (b) Equivalent mother wavelets and spectra for the 4-band\nSteerable Pyramid.\n\n(cid:21)ai and (cid:21)si parameters were constrained to be the same for all orientation bands and\nwere adapted over many images with (cid:21)N (cid:12)xed. Shown in (cid:12)gure 2 are the learned\n i(x; y) which parameterize W , with their corresponding 2D spectra. Three di(cid:11)er-\nent degrees of overcompleteness were tested. The results are shown for 2 band, 4\nband and 6 band wavelet bases. As the degree of overcompleteness increases, the\nresulting functions show tighter tuning to orientation. The basis (cid:12)lters for a 4 band\nSteerable Pyramid [10] are also shown for comparison, to illustrate the similarity\nto the learned functions.\n\nlearned\nsteer\n\n)\n\nB\nd\n(\n \n\nR\nN\nS\n\n27\n\n26.5\n\n26\n\n25.5\n\n25\n\n24.5\n\n24\n\n23.5\n\n23\n\n22.5\n\n1.0\n\n2.0\n\n3.0\n\n4.0\n\n5.0\n\n% nonzeros\n\nFigure 3: Sparsity comparison between the learned basis (top) and the steerable\nbasis (bottom). The y axis represents the signal-to-noise ratio (SNR) in dB achieved\nfor each method for a given percentage of nonzeros.\n\n5.1 Sparsity\n\nWe evaluated the sparsity of the representations obtained with the four band learned\nfunctions and the sampling method with those obtained using the same sampling\nmethod and the four band Steerable Pyramid (cid:12)lters [10]. In order to explore the\nSNR curves for each basis, we used a variety of values for (cid:21)s so as to obtain di(cid:11)erent\nlevels of sparsity. The same images were used for both bases. The results are given\nin (cid:12)gure 3. Each dot on the line represents a di(cid:11)erent value of (cid:21)s. The results were\nsimilar, with the learned basis yielding slightly higher SNR (about 0.5 dB) for the\nsame number of active coe(cid:14)cients.\n\n\f5.2 Denoising\n\nWe evaluated our inference method and learned basis functions by denoising images\ncontaining known amounts of additive i.i.d. Gaussian noise. Denoising was accom-\nplished by averaging samples taken from the posterior distribution for each image\nvia Gibbs sampling to approximate the posterior mean. Gibbs sampling was per-\nformed on a four level pyramid using the 6 band learned wavelet basis, and also\nusing the 6 band Steerable basis. The (cid:21)N ; (cid:21)si and (cid:21)ai parameters were adapted to\neach noisy image during sampling for blind denoising in which the noise variance\nwas assumed to be unknown. We compared these results to the wiener2 function in\nMATLAB, and also to BayesCore [9], a Bayesian method for computing an optimal\nsoft thresholding, or coring, function for a generalized Laplacian prior. For wiener2,\nthe best neighborhood size was used for each image. Table 1 gives the SNR results\nfor each method when applied to some standard test images for three di(cid:11)erent lev-\nels of i.i.d. Gaussian noise with standard deviation (cid:27). Figure 4 shows a cropped\nsubregion of the results for the \\Einstein\" image with (cid:27) = 10.\n\n6 Summary and Conclusions\n\nWe have shown that a wavelet basis and a mixture prior composed of a Dirac delta\nfunction and a Gaussian can be adapted to natural images resulting in very sparse\nimage representations. The resulting basis is very similar to a Steerable basis, both\nin appearance and sparsity of the resulting image representations. It appears that\nthe Steerable basis may be nearly optimal for producing sparse representations of\nnatural scenes. Denoising results indicate that using a sparse prior and an inference\nmethod to properly account for the non-orthogonality of the representation may\nyield a signi(cid:12)cant improvement over wavelet coring methods that use (cid:12)ltered coe(cid:14)-\ncients. More work needs to be done to determine whether the coding gains achieved\nare due to the choice of prior versus the basis or inference/estimation method used.\n\nSupported by NIMH R29-MH057921. Phil Sallee\u2019s work was\nAcknowledgments\nalso supported in part by a United States Department of Education Government\nAssistance in Areas of National Need (DOE-GAANN) grant #P200A980307.\n\nImage\nEinstein\n\nLena\n\nGoldhill\n\nFruit\n\nnoise level\n\n(cid:27) = 10\n(cid:27) = 20\n(cid:27) = 30\n(cid:27) = 10\n(cid:27) = 20\n(cid:27) = 30\n(cid:27) = 10\n(cid:27) = 20\n(cid:27) = 30\n(cid:27) = 10\n(cid:27) = 20\n(cid:27) = 30\n\nnoisy wiener2 BayesCore S6\n16.36\n12.40\n13.44\n6.40\n2.89\n11.81\n19.91\n13.61\n16.88\n7.59\n14.99\n4.07\n18.14\n13.86\n7.83\n15.18\n13.61\n4.28\n22.09\n16.25\n18.97\n10.24\n6.70\n17.21\n\n15.80\n12.61\n10.95\n19.05\n15.51\n13.25\n17.56\n14.32\n12.64\n21.87\n18.15\n15.97\n\nD+G S6\n16.47\n13.80\n12.28\n20.37\n17.46\n15.48\n18.10\n15.41\n13.92\n22.78\n19.61\n17.72\n\nD+G L6\n16.19\n13.79\n12.29\n20.21\n17.54\n15.55\n17.90\n15.41\n13.95\n22.38\n19.42\n17.66\n\nTable 1: SNR values (in dB) for noisy and denoised images contaminated with\nadditive i.i.d. Gaussian noise of std.dev. (cid:27). \\D+G\" means Delta-plus-Gaussian\nprior, \\S6\" means 6-Band Steerable basis, and \\L6\" means 6-Band Learned basis.\n\n\foriginal\n\nnoisy (s=10) SNR=12.3983\n\nwiener2 SNR=15.8033\n\nBayesCore steer6 SNR=16.3591\n\nD+G steer6 SNR=16.4714\n\nD+G learned6 SNR=16.1939\n\nFigure 4: Denoising example. A cropped subregion of the Einstein image and\ndenoised images for each noise reduction method for noise std.dev. (cid:27)=10.\n\nReferences\n\n[1] Abromovich F, Sapatinas T, Silverman B (1996), Wavelet Thresholding via a Bayesian\n\nApproach, preprint.\n\n[2] Chipman H, Kolaczyk E, McCulloch R (1997) Adaptive bayesian wavelet shrinkage,\n\nJ. Amer. Statist. Assoc. 92(440): 1413-1421.\n\n[3] Chang SG, Yu B, Vetterli M (2000). Spatially Adaptive Wavelet Thresholding with\nContext Modelling for Image Denoising. IEEE Trans. on Image Proc., 9(9): 1522-1531.\n\n[4] Coifman RR, Donoho DL (1995). Translation-invariant de-noising, in Wavelets and\nStatistics, A.Antoniadis and G. Oppenheim, Eds. Berlin, Germany: Springer-Varlag.\n\n[5] Crouse MS, Nowak RD and Baraniuk RG (1998) Wavelet-based Statistical Signal\n\nProcessing using Hidden Markov Models, IEEE Trans. Signal Proc., 46(4): 886-902.\n\n[6] Freeman WT, Adelson EH (1991) The Design and Use of Steerable Filters. IEEE\n\nTrans. Patt. Anal. and Machine Intell., 13(9): 891-906.\n\n[7] Olshausen BA, Sallee P, Lewicki MS (2001) Learning sparse image codes using a\n\nwavelet pyramid architecture, Adv. in Neural Inf. Proc. Sys., 13: 887-893.\n\n[8] Simoncelli EP, Freeman WT, Adelson EH, Heeger DJ (1992) Shiftable multiscale trans-\n\nforms, IEEE Transactions on Information Theory, 38(2): 587-607.\n\n[9] Simoncelli EP, Adelson EH (1996) Noise removal via Bayesian wavelet coring, Pre-\n\nsented at: 3rd IEEE International Conf. on Image Proc., Laussanne Switzerland.\n\n[10] Simoncelli EP, Freeman WT (1995). The Steerable Pyramid: A Flexible Architecture\n\nfor Multi-scale Derivative Computation, IEEE Int. Conf. on Image Processing.\n\n\f", "award": [], "sourceid": 2145, "authors": [{"given_name": "Phil", "family_name": "Sallee", "institution": null}, {"given_name": "Bruno", "family_name": "Olshausen", "institution": null}]}