{"title": "Random Projections for Manifold Learning", "book": "Advances in Neural Information Processing Systems", "page_first": 641, "page_last": 648, "abstract": "We propose a novel method for {\\em linear} dimensionality reduction of manifold modeled data. First, we show that with a small number $M$ of {\\em random projections} of sample points in $\\reals^N$ belonging to an unknown $K$-dimensional Euclidean manifold, the intrinsic dimension (ID) of the sample set can be estimated to high accuracy. Second, we rigorously prove that using only this set of random projections, we can estimate the structure of the underlying manifold. In both cases, the number random projections required is linear in $K$ and logarithmic in $N$, meaning that $K 50. (right)ML-RPonthehandrotationdatabase(N = 3840). For\nM > 60,theIsomapvarianceisindistinguishablefromthevarianceobtainedintheambientspace.\n\nsion of the underlying manifold. We plot the intrinsic dimension of the dataset against the minimum\n\nnumber of projections required such that bK\u03a6 is within 10% of the conventional GP estimate bK (this\n\nis equivalent to choosing \u03b4 = 0.1 in Theorem 3.1). We observe the predicted linearity (Theorem 3.1)\nin the variation of M vs K.\nFinally, we turn our attention to two common datasets (Figure 3) found in the literature on dimension\nestimation \u2013 the face database2 [6], and the hand rotation database [17].3 The face database is a\ncollection of 698 arti\ufb01cial snapshots of a face (N = 64 \u00d7 64 = 4096) varying under 3 degrees of\nfreedom: 2 angles for pose and 1 for lighting dimension. The signals are therefore believed to reside\non a 3D manifold in an ambient space of dimension 4096. The hand rotation database is a set of\n90 images (N = 64 \u00d7 60 = 3840) of rotations of a hand holding an object. Although the image\nappearance manifold is ostensibly one-dimensional, estimators in the literature always overestimate\nits ID [11].\n\nRandom projections of each sample in the databases were obtained by computing the inner product\nof the image samples with an increasing number of rows of the random orthoprojector \u03a6. We\nnote that in the case of the face database, for M > 60, the Isomap variance on the randomly\nprojected points closely approximates the variance obtained with full image data. This behavior of\nconvergence of the variance to the best possible value is even more sharply observed in the hand\nrotation database, in which the two variance curves are indistinguishable for M > 60. These results\nare particularly encouraging and demonstrate the validity of the claims made in Section 3.\n\n6 Discussion\n\nOur main theoretical contributions in this paper are the explicit values for the lower bounds on the\nminimum number of random projections required to perform ID estimation and subsequent manifold\nlearning using Isomap, with high guaranteed accuracy levels. We also developed an empirical greedy\nalgorithm (ML-RP) for practical situations. Experiments on simple cases, such as uniformly gener-\nated hyperspheres of varying dimension, and more complex situations, such as the image databases\ndisplayed in Figure 3, provide suf\ufb01cient evidence of the nature of the bounds described above.\n\n2http://isomap.stanford.edu\n3http://vasc.ri.cmu.edu//idb/html/motion/hand/index.html. Note that we use a subsampled version of the\n\ndatabase used in the literature, both in terms of resolution of the image and sampling of the manifold.\n\n\fThe method of random projections is thus a powerful tool for ensuring the stable embedding of low-\ndimensional manifolds into an intermediate space of reasonable size. The motivation for developing\nresults and algorithms that involve random measurements of high-dimensional data is signi\ufb01cant,\nparticularly due to the increasing attention that Compressive Sensing (CS) has received recently. It\nis now possible to think of settings involving a huge number of low-power devices that inexpen-\nsively capture, store, and transmit a very small number of measurements of high-dimensional data.\nML-RP is applicable in all such situations. In situations where the bottleneck lies in the transmission\nof the data to the central processing node, ML-RP provides a simple solution to the manifold learn-\ning problem and ensures that with minimum transmitted amount of information, effective manifold\nlearning can be performed. The metric structure of the projected dataset upon termination of ML-\nRP closely resembles that of the original dataset with high probability; thus, ML-RP can be viewed\nas a novel adaptive algorithm for \ufb01nding an ef\ufb01cient, reduced representation of data of very large\ndimension.\n\nReferences\n\n[1] R. G. Baraniuk and M. B. Wakin. Random projections of smooth manifolds. 2007. To appear\n\nin Foundations of Computational Mathematics.\n\n[2] M. B. Wakin, J. N. Laska, M. F. Duarte, D. Baron, S. Sarvotham, D. Takhar, K. F. Kelly, and\nR. G. Baraniuk. An architecture for compressive imaging. In IEEE International Conference\non Image Processing (ICIP), pages 1273\u20131276, Oct. 2006.\n\n[3] S. Kirolos, J.N. Laska, M.B. Wakin, M.F. Duarte, D.Baron, T. Ragheb, Y. Massoud, and R.G.\nBaraniuk. Analog-to-information conversion via random demodulation. In Proc. IEEE Dallas\nCircuits and Systems Workshop (DCAS), 2006.\n\n[4] E. J. Cand`es, J. Romberg, and T. Tao. Robust uncertainty principles: Exact signal reconstruc-\ntion from highly incomplete frequency information. IEEE Trans. Info. Theory, 52(2):489\u2013509,\nFeb. 2006.\n\n[5] D. L. Donoho. Compressed sensing. IEEE Trans. Info. Theory, 52(4):1289\u20131306, September\n\n2006.\n\n[6] J. B. Tenenbaum, V.de Silva, and J. C. Landford. A global geometric framework for nonlinear\n\ndimensionality reduction. Science, 290:2319\u20132323, 2000.\n\n[7] P. Grassberger and I. Procaccia. Measuring the strangeness of strange attractors. Physica D\n\nNonlinear Phenomena, 9:189\u2013208, 1983.\n\n[8] J. Theiler. Statistical precision of dimension estimators. Physical Review A, 41(6):3038\u20133051,\n\n1990.\n\n[9] F. Camastra. Data dimensionality estimation methods: a survey. Pattern Recognition, 36:2945\u2013\n\n2954, 2003.\n\n[10] J. A. Costa and A. O. Hero. Geodesic entropic graphs for dimension and entropy estimation in\n\nmanifold learning. IEEE Trans. Signal Processing, 52(8):2210\u20132221, August 2004.\n\n[11] E. Levina and P. J. Bickel. Maximum likelihood estimation of intrinsic dimension. In Advances\n\nin NIPS, volume 17. MIT Press, 2005.\n\n[12] S. Roweis and L. Saul. Nonlinear dimensionality reduction by locally linear embedding. Sci-\n\nence, 290:2323\u20132326, 2000.\n\n[13] D. Donoho and C. Grimes. Hessian eigenmaps: locally linear embedding techniques for high\n\ndimensional data. Proc. of National Academy of Sciences, 100(10):5591\u20135596, 2003.\n\n[14] Sanjoy Dasgupta and Anupam Gupta. An elementary proof of the JL lemma. Technical Report\n\nTR-99-006, University of California, Berkeley, 1999.\n\n[15] C. Hegde, M. B. Wakin, and R. G. Baraniuk. Random projections for manifold learning -\n\nproofs and analysis. Technical Report TREE 0710, Rice University, 2007.\n\n[16] M. Bernstein, V. de Silva, J. Langford, and J. Tenenbaum. Graph approximations to geodesics\n\non embedded manifolds, 2000. Technical report, Stanford University.\n\n[17] B. K\u00b4egl. Intrinsic dimension estimation using packing numbers. In Advances in NIPS, vol-\n\nume 14. MIT Press, 2002.\n\n\f", "award": [], "sourceid": 1100, "authors": [{"given_name": "Chinmay", "family_name": "Hegde", "institution": null}, {"given_name": "Michael", "family_name": "Wakin", "institution": null}, {"given_name": "Richard", "family_name": "Baraniuk", "institution": null}]}