{"title": "Which Space Partitioning Tree to Use for Search?", "book": "Advances in Neural Information Processing Systems", "page_first": 656, "page_last": 664, "abstract": "We consider the task of nearest-neighbor search with the class of binary-space-partitioning trees, which includes kd-trees, principal axis trees and random projection trees, and try to rigorously answer the question which tree to use for nearest-neighbor search?'' To this end, we present the theoretical results which imply that trees with better vector quantization performance have better search performance guarantees. We also explore another factor affecting the search performance -- margins of the partitions in these trees. We demonstrate, both theoretically and empirically, that large margin partitions can improve the search performance of a space-partitioning tree. \"", "full_text": "Which Space Partitioning Tree to Use for Search?\n\nP. Ram\n\nGeorgia Tech. / Skytree, Inc.\n\nAtlanta, GA 30308\n\np.ram@gatech.edu\n\nA. G. Gray\nGeorgia Tech.\n\nAtlanta, GA 30308\n\nagray@cc.gatech.edu\n\nAbstract\n\nWe consider the task of nearest-neighbor search with the class of binary-space-\npartitioning trees, which includes kd-trees, principal axis trees and random projec-\ntion trees, and try to rigorously answer the question \u201cwhich tree to use for nearest-\nneighbor search?\u201d To this end, we present the theoretical results which imply that\ntrees with better vector quantization performance have better search performance\nguarantees. We also explore another factor affecting the search performance \u2013\nmargins of the partitions in these trees. We demonstrate, both theoretically and\nempirically, that large margin partitions can improve tree search performance.\n\n1 Nearest-neighbor search\n\nNearest-neighbor search is ubiquitous in computer science. Several techniques exist for nearest-\nneighbor search, but most algorithms can be categorized into two following groups based on the in-\ndexing scheme used \u2013 (1) search with hierarchical tree indices, or (2) search with hash-based indices.\nAlthough multidimensional binary space-partitioning trees (or BSP-trees), such as kd-trees [1], are\nwidely used for nearest-neighbor search, it is believed that their performances degrade with increas-\ning dimensions. Standard worst-case analyses of search with BSP-trees in high dimensions usually\nlead to trivial guarantees (such as, an \u2126(n) search time guarantee for a single nearest-neighbor query\nin a set of n points). This is generally attributed to the \u201ccurse of dimensionality\u201d \u2013 in the worst case,\nthe high dimensionality can force the search algorithm to visit every node in the BSP-tree.\nHowever, these BSP-trees are very simple and intuitive, and still used in practice with success.\nThe occasional favorable performances of BSP-trees in high dimensions are attributed to the low\n\u201cintrinsic\u201d dimensionality of real data. However, no clear relationship between the BSP-tree search\nperformance and the intrinsic data properties is known. We present theoretical results which link the\nsearch performance of BSP-trees to properties of the data and the tree. This allows us to identify\nimplicit factors in\ufb02uencing BSP-tree search performance \u2014 knowing these driving factors allows\nus to develop successful heuristics for BSP-trees with improved search performance.\nEach node in a BSP-tree represents a region of the space and\neach non-leaf node has a left and right child representing a dis-\njoint partition of this region with some separating hyperplane\nand threshold (w, b). A search query on this tree is usually\nanswered with a depth-\ufb01rst branch-and-bound algorithm. Al-\ngorithm 1 presents a simpli\ufb01ed version where a search query\nis answered with a small set of neighbor candidates of any de-\nsired size by performing a greedy depth-\ufb01rst tree traversal to\na speci\ufb01ed depth. This is known as defeatist tree search. We\nare not aware of any data-dependent analysis of the quality of\nthe results from defeatist BSP-tree search. However, Verma et\nal. (2009) [2] presented adaptive data-dependent analyses of\nsome BSP-trees for the task of vector quantization. These re-\nsults show precise connections between the quantization per-\nformance of the BSP-trees and certain properties of the data\n(we will present these data properties in Section 2).\n\nQuery q, Desired depth l\nOutput: Candidate neighbor p\ncurrent tree depth lc \u2190 0\ncurrent tree node Tc \u2190 T\nwhile lc < l do\n\nend if\nIncrement depth lc \u2190 lc + 1\nend while\np \u2190 arg minr\u2208Tc\u2229S (cid:107)q \u2212 r(cid:107).\n\nif (cid:104)Tc.w, q(cid:105) + Tc.b \u2264 0 then\nelse\n\nTc \u2190 Tc.left child\nTc \u2190 Tc.right child\n\nAlgorithm 1 BSP-tree search\nInput: BSP-tree T on set S,\n\n1\n\n\f(a) kd-tree\n\n(b) RP-tree\n\n(c) MM-tree\n\nFigure 1: Binary space-partitioning trees.\n\nWe establish search performance guarantees for BSP-trees by linking their nearest-neighbor perfor-\nmance to their vector quantization performance and utilizing the recent guarantees on the BSP-tree\nvector quantization. Our results provide theoretical evidence, for the \ufb01rst time, that better quantiza-\ntion performance implies better search performance1. These results also motivate the use of large\nmargin BSP-trees, trees that hierarchically partition the data with a large (geometric) margin, for\nbetter nearest-neighbor search performance. After discussing some existing literature on nearest-\nneighbor search and vector quantization in Section 2, we discuss our following contributions:\n\u2022 We present performance guarantees for Algorithm 1 in Section 3, linking search performance\nto vector quantization performance. Speci\ufb01cally, we show that for any balanced BSP-tree and a\ndepth l, under some conditions, the worst-case search error incurred by the neighbor candidate\nreturned by Algorithm 1 is proportional to a factor which is\n\n(cid:18) 2l/2 exp(\u2212l/2\u03b2)\n\n(n/2l)1/O(d) \u2212 2\n\nO\n\n(cid:19)\n\n,\n\nproportional to (1/\u03b3) where \u03b3 is the smallest margin size of all the partitions in T .\n\nwhere \u03b2 corresponds to the quantization performance of the tree (smaller \u03b2 implies smaller\nquantization error) and d is closely related to the doubling dimension of the dataset (as opposed\nto the ambient dimension D of the dataset). This implies that better quantization produces better\nworst-case search results. Moreover, this result implies that smaller l produces improved worst-\ncase performance (smaller l does imply more computation, hence it is intuitive to expect less\nerror at the cost of computation). Finally, there is also the expected dependence on the intrinsic\ndimensionality d \u2013 increasing d implies deteriorating worst-case performance. The theoretical\nresults are empirically veri\ufb01ed in this section as well.\n\u2022 In Section 3, we also show that the worst-case search error for Algorithm 1 with a BSP-tree T is\n\u2022 We present the quantization performance guarantee of a large margin BSP tree in Section 4.\nThese results indicate that for a given dataset, the best BSP-tree for search is the one with the best\ncombination of low quantization error and large partition margins. We conclude with this insight\nand related unanswered questions in Section 5.\n2 Search and vector quantization\nBinary space-partitioning trees (or BSP-trees) are hierarchical data structures providing a multi-\nresolution view of the dataset indexed. There are several space-partitioning heuristics for a BSP-\ntree construction. A tree is constructed by recursively applying a heuristic partition. The most\npopular kd-tree uses axis-aligned partitions (Figure 1(a)), often employing a median split along the\ncoordinate axis of the data in the tree node with the largest spread. The principal-axis tree (PA-tree)\npartitions the space at each node at the median along the principal eigenvector of the covariance\nmatrix of the data in that node [3, 4]. Another heuristic partitions the space based on a 2-means\nclustering of the data in the node to form the two-means tree (2M-tree) [5, 6]. The random-projection\ntree (RP-tree) partitions the space by projecting the data along a random standard normal direction\nand choosing an appropriate splitting threshold [7] (Figure 1(b)). The max-margin tree (MM-tree) is\nbuilt by recursively employing large margin partitions of the data [8] (Figure 1(c)). The unsupervised\nlarge margin splits are usually performed using max-margin clustering techniques [9].\nSearch. Nearest-neighbor search with a BSP-tree usually involves a depth-\ufb01rst branch-and-bound\nalgorithm which guarantees the search approximation (exact search is a special case of approximate\nsearch with zero approximation) by a depth-\ufb01rst traversal of the tree followed by a backtrack up the\ntree as required. This makes the tree traversal unpredictable leading to trivial worst-case runtime\n\n1This intuitive connection is widely believed but never rigorously established to the best of our knowledge.\n\n2\n\n\fguarantees. On the other hand, locality-sensitive hashing [10] based methods approach search in a\ndifferent way. After indexing the dataset into hash tables, a query is answered by selecting candidate\npoints from these hash tables. The candidate set size implies the worst-case search time bound. The\nhash table construction guarantees the set size and search approximation. Algorithm 1 uses a BSP-\ntree to select a candidate set for a query with defeatist tree search. For a balanced tree on n points,\nthe candidate set size at depth l is n/2l and the search runtime is O(l + n/2l), with l \u2264 log2 n. For\nany choice of the depth l, we present the \ufb01rst approximation guarantee for this search process.\nDefeatist BSP-tree search has been explored with the spill tree [11], a binary tree with overlapping\nsibling nodes unlike the disjoint nodes in the usual BSP-tree. The search involves selecting the can-\ndidates in (all) the leaf node(s) which contain the query. The level of overlap guarantees the search\napproximation, but this search method lacks any rigorous runtime guarantee; it is hard to bound the\nnumber of leaf nodes that might contain any given query. Dasgupta & Sinha (2013) [12] show that\nthe probability of \ufb01nding the exact nearest neighbor with defeatist search on certain randomized\npartition trees (randomized spill trees and RP-trees being among them) is directly proportional to\nthe relative contrast of the search task [13], a recently proposed quantity which characterizes the\ndif\ufb01culty of a search problem (lower relative contrast makes exact search harder).\nVector Quantization. Recent work by Verma et al., 2009 [2] has established theoretical guarantees\nfor some of these BSP-trees for the task of vector quantization. Given a set of points S \u2282 RD of n\npoints, the task of vector quantization is to generate a set of points M \u2282 RD of size k (cid:28) n with\nlow average quantization error. The optimal quantizer for any region A is given by the mean \u00b5(A)\nof the data points lying in that region. The quantization error of the region A is then given by\n\nVS(A) =\n\n1\n\n|A \u2229 S|\n\n(cid:107)x \u2212 \u00b5(A)(cid:107)2\n2 ,\n\n(1)\n\n(cid:88)\n\nx\u2208A\u2229S\n\n(cid:17)\n(cid:17)\n\nand the average quantization error of a disjoint partition of region A into Al and Ar is given by:\n\nVS({Al, Ar}) = (|Al \u2229 S|VS(Al) + |Ar \u2229 S|VS(Ar)) /|A \u2229 S|.\n\n(2)\nTree-based structured vector quantization is used for ef\ufb01cient vector quantization \u2013 a BSP-tree of\ndepth log2 k partitions the space containing S into k disjoint regions to produce a k-quantization of\nS. The theoretical results for tree-based vector quantization guarantee the improvement in average\nquantization error obtained by partitioning any single region (with a single quantizer) into two dis-\njoints regions (with two quantizers) in the following form (introduced by Freund et al. (2007) [14]):\nDe\ufb01nition 2.1. For a set S \u2282 RD, a region A par-\ntitioned into two disjoint regions {Al, Ar}, and a\ndata-dependent quantity \u03b2 > 1, the quantization er-\nror improvement is characterized by:\n\n/\u03bb1\n\n.\n=\n\nTree\nPA-tree\nRP-tree\nkd-tree\n2M-tree\nMM-tree\u2217 O(\u03c1) : \u03c1\n\nDe\ufb01nition of \u03b2\nO(\u00012) : \u0001\nO(dc)\n\u00d7\noptimal (smallest possible)\n/\u03b32\n\ni=1 \u03bbi\n\n.\n=\n\n(cid:16)(cid:80)D\n(cid:16)(cid:80)D\n\nVS({Al, Ar}) < (1 \u2212 1/\u03b2)VS(A).\n\n(3)\n\ni=1 \u03bbi\n\nThe quantization performance depends inversely on\nthe data-dependent quantity \u03b2 \u2013 lower \u03b2 implies bet-\nter quantization. We present the de\ufb01nition of \u03b2 for\ndifferent BSP-trees in Table 1. For the PA-tree, \u03b2\ndepends on the ratio of the sum of the eigenval-\nues of the covariance matrix of data (A \u2229 S) to the\nprincipal eigenvalue. The improvement rate \u03b2 for\nthe RP-tree depends on the covariance dimension\nof the data in the node A (\u03b2 = O(dc)) [7], which\nroughly corresponds to the lowest dimensionality of\nan af\ufb01ne plane that captures most of the data covari-\nance. The 2M-tree does not have an explicit \u03b2 but\nit has the optimal theoretical improvement rate for a\nsingle partition because the 2-means clustering ob-\njective is equal to |Al|V(Al) + |Ar|V(Ar) and minimizing this objective maximizes \u03b2. The 2-\nmeans problem is NP-hard and an approximate solution is used in practice. These theoretical re-\nsults are valid under the condition that there are no outliers in A \u2229 S. This is characterized as\nmaxx,y\u2208A\u2229S (cid:107)x \u2212 y(cid:107)2 \u2264 \u03b7VS(A) for a \ufb01xed \u03b7 > 0. This notion of the absence of outliers was\n\ufb01rst introduced for the theoretical analysis of the RP-trees [7]. Verma et al. (2009) [2] describe\noutliers as \u201cpoints that are much farther away from the mean than the typical distance-from-mean\u201d.\nIn this situation, an alternate type of partition is used to remove these outliers that are farther away\n\nTable 1: \u03b2 for various trees. \u03bb1, . . . , \u03bbD are\nthe sorted eigenvalues of the covariance matrix\nof A \u2229 S in descending order, and dc < D is\nthe covariance dimension of A \u2229 S. The results\nfor PA-tree and 2M-tree are due to Verma et al.\n(2009) [2]. The PA-tree result can be improved to\nO(\u0001) from O(\u00012) with an additional assumption\n[2]. The RP-tree result is in Freund et al. (2007)\n[14], which also has the precise de\ufb01nition of dc.\nWe establish the result for MM-tree in Section 4.\n\u03b3 is the margin size of the large margin partition.\nNo such guarantee for kd-trees is known to us.\n\n3\n\n\ffrom the mean than expected. For \u03b7 \u2265 8, this alternate partitioning is guaranteed to reduce the data\ndiameter (maxx,y\u2208A\u2229S (cid:107)x \u2212 y(cid:107)) of the resulting nodes by a constant fraction [7, Lemma 12], and\ncan be used until a region contain no outliers, at which point, the usual hyperplane partition can be\nused with their respective theoretical quantization guarantees. The implicit assumption is that the\nalternate partitioning scheme is employed rarely.\nThese results for BSP-tree quantization performance indicate that different heuristics are adaptive\nto different properties of the data. However, no existing theoretical result relates this performance\nof BSP-trees to their search performance. Making the precise connection between the quantization\nperformance and the search performance of these BSP-trees is a contribution of this paper.\n3 Approximation guarantees for BSP-tree search\nIn this section, we formally present the data and tree dependent performance guarantees on the\nsearch with BSP-trees using Algorithm 1. The quality of nearest-neighbor search can be quantized\nin two ways \u2013 (i) distance error and (ii) rank of the candidate neighbor. We present guarantees for\nboth notions of search error2. For a query q and a set of points S and a neighbor candidate p \u2208 S,\ndistance error \u0001(q) =\n\nminr\u2208S(cid:107)q\u2212r(cid:107) \u2212 1, and rank \u03c4 (q) = |{r \u2208 S : (cid:107)q \u2212 r(cid:107) < (cid:107)q \u2212 p(cid:107)}| + 1.\n\n(cid:107)q\u2212p(cid:107)\n\nAlgorithm 1 requires the query traversal depth l as an input. The search runtime is O(l + (n/2l)).\nThe depth can be chosen based on the desired runtime. Equivalently, the depth can be chosen based\non the desired number of candidates m; for a balanced binary tree on a dataset S of n points with leaf\nnodes containing a single point, the appropriate depth l = log2 n \u2212 (cid:100)log2 m(cid:101). We will be building\non the existing results on vector quantization error [2] to present the worst case error guarantee for\nAlgorithm 1. We need the following de\ufb01nitions to precisely state our results:\nDe\ufb01nition 3.1. An \u03c9-balanced split partitioning a region A into disjoint regions {A1, A2} implies\n||A1 \u2229 S| \u2212 |A2 \u2229 S|| \u2264 \u03c9|A \u2229 S|.\nFor a balanced tree corresponding to recursive median splits, such as the PA-tree and the kd-tree,\n\u03c9 \u2248 0. Non-zero values of \u03c9 (cid:28) 1, corresponding to approximately balanced trees, allow us to\npotentially adapt better to some structure in the data at the cost of slightly losing the tree balance.\nFor the MM-tree (discussed in detail in Section 4), \u03c9-balanced splits are enforced for any speci\ufb01ed\nvalue of \u03c9. Approximately balanced trees have a depth bound of O(log n) [8, Theorem 3.1]. For\n. For the\na tree with \u03c9-balanced splits, the worst case runtime of Algorithm 1 is O\n2M-tree, \u03c9-balanced splits are not enforced. Hence the actual value of \u03c9 could be high for a 2M-tree.\n(p, \u2206) = {r \u2208 S : (cid:107)p \u2212 r(cid:107) < \u2206} denote the points in S contained in a ball\nDe\ufb01nition 3.2. Let B(cid:96)2\nof radius \u2206 around some p \u2208 S with respect to the (cid:96)2 metric. The expansion constant of (S, (cid:96)2) is\n\n(cid:1)l\nl +(cid:0) 1+\u03c9\n(p, \u2206)(cid:12)(cid:12) \u2200p \u2208 S and \u2200\u2206 > 0.\n\nde\ufb01ned as the smallest c \u2265 2 such(cid:12)(cid:12)B(cid:96)2\n\n(p, 2\u2206)(cid:12)(cid:12) \u2264 c(cid:12)(cid:12)B(cid:96)2\n\n(cid:16)\n\n(cid:17)\n\nn\n\n2\n\nBounded expansion constants correspond to growth-restricted metrics [15]. The expansion constant\ncharacterizes the data distribution, and c \u223c 2O(d) where d is the doubling dimension of the set S\n(cid:80)\nwith respect to the (cid:96)2 metric. The relationship is exact for points on a D-dimensional grid (i.e.,\nc = \u0398(2D)). Equipped with these de\ufb01nitions, we have the following guarantee for Algorithm 1:\nx,y\u2208S (cid:107)x \u2212 y(cid:107)2, the BSP\nTheorem 3.1. Consider a dataset S \u2282 RD of n points with \u03c8 = 1\ntree T built on S and a query q \u2208 RD with the following conditions :\n\n(C2) Let T be complete till a depth L <(cid:0)log2\n\n(C1) Let (A \u2229 (S \u222a {q}), (cid:96)2) have an expansion constant at most \u02dcc for any convex set A \u2282 RD.\n(C3) Let \u03b2\u2217 correspond to the worst quantization error improvement rate over all splits in T .\n(C4) For any node A in the tree T , let maxx,y\u2208A\u2229S (cid:107)x \u2212 y(cid:107)2 \u2264 \u03b7VS(A) for a \ufb01xed \u03b7 \u2265 8.\n\n(cid:1) /(1 \u2212 log2(1 \u2212 \u03c9)) with \u03c9-balanced splits.\n\nFor \u03b1 = 1/(1 \u2212 \u03c9), the upper bound du on the distance of q to the neighbor candidate p returned\nby Algorithm 1 with depth l \u2264 L is given by\n\u221a\n2\n\n2n2\n\nn\n\u02dcc\n\n(cid:107)q \u2212 p(cid:107) \u2264 du =\n\n\u03b7\u03c8 \u00b7 (2\u03b1)l/2 \u00b7 exp(\u2212l/2\u03b2\u2217)\n(n/(2\u03b1)l)1/ log2 \u02dcc \u2212 2\n\n.\n\n(4)\n\n2The distance error corresponds to the relative error in terms of the actual distance values. The rank is one\nmore than the number of points in S which are better neighbor candidates than p. The nearest-neighbor of q\nhas rank 1 and distance error 0. The appropriate notion of error depends on the search application.\n\n4\n\n\fneighbor distance is inversely proportional to(cid:0)n/(2\u03b1)l(cid:1)1/ log2 \u02dcc, implying deteriorating bounds du\n\nNow \u03b7 is \ufb01xed, and \u03c8 is \ufb01xed for a dataset S. Then, for a \ufb01xed \u03c9, this result implies that between\ntwo types of BSP-trees on the same set and the same query, Algorithm 1 has a better worst-case guar-\nantee on the candidate-neighbor distance for the tree with better quantization performance (smaller\n\u03b2\u2217). Moreover, for a particular tree with \u03b2\u2217 \u2265 log2 e, du is non-decreasing in l. This is expected\nbecause as we traverse down the tree, we can never reduce the candidate neighbor distance. At the\nroot level (l = 0), the candidate neighbor is the nearest-neighbor. As we descend down the tree,\nthe candidate neighbor distance will worsen if a tree split separates the query from its closer neigh-\nbors. This behavior is implied in Equation (4). For a chosen depth l in Algorithm 1, the candidate\nwith increasing \u02dcc. Since log2 \u02dcc \u223c O(d), larger intrinsic dimensionality implies worse guarantees as\nexpected from the curse of dimensionality. To prove Theorem 3.1, we use the following result:\nLemma 3.1. Under the conditions of Theorem 3.1, for any node A at a depth l in the BSP-tree T\non S, VS(A) \u2264 \u03c8 (2/(1 \u2212 \u03c9))l exp(\u2212l/\u03b2\u2217).\nThis result is obtained by recursively applying the quantization error improvement in De\ufb01nition 2.1\nover l levels of the tree (the proof is in Appendix A).\nProof of Theorem 3.1. Consider the node A at depth l in the tree containing q, and let m = |A \u2229 S|.\nLet D = maxx,y\u2208A\u2229S (cid:107)x \u2212 y(cid:107), let d = minx\u2208A\u2229S (cid:107)q \u2212 x(cid:107), and let B(cid:96)2\n(q, \u2206) = {x \u2208 A \u2229 (S \u222a\n{q}) : (cid:107)q \u2212 x(cid:107) < \u2206}. Then, by the De\ufb01nition 3.2 and condition C1,\n(q, d)| = \u02dcclog2(cid:100) D+d\n\nd (cid:101)|B(cid:96)2\nwhere the equality follows from the fact that B(cid:96)2\nthis above gives us m1/ log2 \u02dcc \u2264 (D/d) + 2. By condition C2, m1/ log2 \u02dcc > 2. Hence we have\n\n(q, D + d)(cid:12)(cid:12) \u2265 m. Using\nd \u2264 D/(m1/ log2 \u02dcc \u2212 2). By construction and condition C4, D \u2264(cid:112)\u03b7VS(A). Now m \u2265 n/(2\u03b1)l.\n\n(q, d) = {q}. Now(cid:12)(cid:12)B(cid:96)2\n\n(q, D + d)(cid:12)(cid:12) \u2264 \u02dcclog2(cid:100) D+d\n\nd (cid:101) \u2264 \u02dcclog2( D+2d\n\n(cid:12)(cid:12)B(cid:96)2\n\n),\n\nd\n\nq )(cid:101), where d\u2217\n\nq = minr\u2208S (cid:107)q \u2212 r(cid:107).\n\n(q, du)| \u2264 \u02dcc(cid:100)log2(du/d\u2217\n\nq)|. Using the fact that |B(cid:96)2\n\nPlugging this above and utilizing Lemma 3.1 gives us the statement of Theorem 3.1.\nNearest-neighbor search error guarantees. Equipped with the bound on the candidate-neighbor\ndistance, we bound the worst-case nearest-neighbor search errors as follows:\nCorollary 3.1. Under the conditions of Theorem 3.1, for any query q at a desired depth l \u2264 L\nq) \u2212 1, and the rank \u03c4 (q) is\nin Algorithm 1, the distance error \u0001(q) is bounded as \u0001(q) \u2264 (du/d\u2217\nbounded as \u03c4 (q) \u2264 \u02dcc(cid:100)log2(du/d\u2217\nProof. The distance error bound follows from the de\ufb01nition of distance error. Let R = {r \u2208\nS : (cid:107)q \u2212 r(cid:107) < du}. By de\ufb01nition, \u03c4 (q) \u2264 |R| + 1. Let B(cid:96)2\n(q, \u2206) = {x \u2208 (S \u222a {q}) : (cid:107)q \u2212 x(cid:107) <\n(q, du)| = |R| + 1 \u2265 \u03c4 (q). From De\ufb01nition\n\u2206}. Since B(cid:96)2\nq)| =\n3.2 and Condition C1, |B(cid:96)2\n(q, d\u2217\n|{q}| = 1 gives us the upper bound on \u03c4 (q).\nThe upper bounds on both forms of search error are directly proportional to du. Hence, the BSP-\ntree with better quantization performance has better search performance guarantees, and increasing\ntraversal depth l implies less computation but worse performance guarantees. Any dependence of\nthis approximation guarantee on the ambient data dimensionality is subsumed by the dependence\non \u03b2\u2217 and \u02dcc. While our result bounds the worst-case performance of Algorithm 1, an average case\nperformance guarantee on the distance error is given by Eq \u0001(q) \u2264 du Eq\n\n(q, du) contains q and R, and q /\u2208 S, |B(cid:96)2\nq )(cid:101)|B(cid:96)2\n\n(cid:1)\u22121, and on the rank\n\nq)(cid:17)\nis given by Eq \u03c4 (q) \u2264 \u02dcc(cid:100)log2 du(cid:101)(cid:16)Eq c\u2212(log2 d\u2217\n\n, since the expectation is over the queries q and du\ndoes not depend on q. For the purposes of relative comparison among BSP-trees, the bounds on the\nexpected error depend solely on du since the term within the expectation over q is tree independent.\nDependence of the nearest-neighbor search error on the partition margins. The search error\nbounds in Corollary 3.1 depend on the true nearest-neighbor distance d\u2217\nq of any query q of which we\nhave no prior knowledge. However, if we partition the data with a large margin split, then we can\nsay that either the candidate neighbor is the true nearest-neighbor of q or that d\u2217\nq is greater than the\nsize of the margin. We characterize the in\ufb02uence of the margin size with the following result:\nCorollary 3.2. Consider the conditions of Theorem 3.1 and a query q at a depth l \u2264 L in Algorithm\n1. Further assume that \u03b3 is the smallest margin size on both sides of any partition in the tree T . Then\nthe distance error is bounded as \u0001(q) \u2264 du/\u03b3 \u2212 1, and the rank is bounded as \u03c4 (q) \u2264 \u02dcc(cid:100)log2(du/\u03b3)(cid:101).\nThis result indicates that if the split margins in a BSP-tree can be increased without adversely affect-\ning its quantization performance, the BSP-tree will have improved nearest-neighbor error guarantees\n\n(q, d\u2217\n\n(cid:0)1/d\u2217\n\nq\n\n5\n\n\ffor the Algorithm 1. This motivated us to consider the max-margin tree [8], a BSP-tree that explicitly\nmaximizes the margin of the split for every split in the tree.\nExplanation of the conditions in Theorem 3.1. Condition C1 implies that for any convex set\nA \u2282 RD, ((A \u2229 (S \u222a {q})), (cid:96)2) has an expansion constant at most \u02dcc. A bounded \u02dcc implies that no\nsubset of (S \u222a {q}), contained in a convex set, has a very high expansion constant. This condition\nimplies that ((S \u222a{q}), (cid:96)2) also has an expansion constant at most \u02dcc (since (S \u222a{q}) is contained in\nits convex hull). However, if (S \u222a {q}, (cid:96)2) has an expansion constant c, this does not imply that the\ndata lying within any convex set has an expansion constant at most c. Hence a bounded expansion\nconstant assumption for (A\u2229(S\u222a{q}), (cid:96)2) for every convex set A \u2282 RD is stronger than a bounded\nexpansion constant assumption for (S \u222a {q}, (cid:96)2)3. Condition C2 ensures that the tree is complete\nso that for every query q and a depth l \u2264 L, there exists a large enough tree node which contains q.\nCondition C3 gives us the worst quantization error improvement rate over all the splits in the tree.\nCondition C4 implies that the squared data diameter of any node A (maxx,y\u2208A\u2229S (cid:107)x \u2212 y(cid:107)2) is\nwithin a constant factor of its quantization error VS(A). This refers to the assumption that the node\nA contains no outliers as described in Section 3 and only hyperplane partitions are used and their\nrespective quantization improvement guarantees presented in Section 2 (Table 1) hold. By placing\ncondition C4, we ignore the alternate partitioning scheme used to remove outliers for simplicity\nof analysis.\nIf we allow a small fraction of the partitions in the tree to be this alternate split, a\nsimilar result can be obtained since the alternate split is the same for all BSP-tree. For two different\nkinds of hyperplane splits, if alternate split is invoked the same number of times in the tree, the\ndifference in their worst-case guarantees for both the trees would again be governed by their worst-\ncase quantization performance (\u03b2\u2217). However, for any \ufb01xed \u03b7, a harder question is whether one\ntype of hyperplane partition violates the inlier condition more often than another type of partition,\nresulting in more alternate partitions. And we do not yet have a theoretical answer for this4.\nEmpirical validation. We examine our theoretical results with 4 datasets \u2013 OPTDIGITS (D = 64,\nn = 3823, 1797 queries), TINY IMAGES (D = 384, n = 5000, 1000 queries), MNIST (D =\n784, n = 6000, 1000 queries), IMAGES (D = 4096, n = 500, 150 queries). We consider the\nfollowing BSP-trees: kd-tree, random-projection (RP) tree, principal axis (PA) tree, two-means (2M)\ntree and max-margin (MM) tree. We only use hyperplane partitions for the tree construction. This is\nS(A) > \u03b7VS(A)) can be computationally\nbecause, \ufb01rstly, the check for the presence of outliers (\u22062\nexpensive for large n, and, secondly, the alternate partition is mostly for the purposes of obtaining\ntheoretical guarantees. The implementation details for the different tree constructions are presented\nin Appendix C. The performance of these BSP-trees are presented in Figure 2. Trees with missing\ndata points for higher depth levels (for example, kd-tree in Figure 2(a) and 2M-tree in Figures 2 (b)\n& (c)) imply that we were unable to grow complete BSP-trees beyond that depth.\nThe quantization performance of the 2M-tree, PA-tree and MM-tree are signi\ufb01cantly better than the\nperformance of the kd-tree and RP-tree and, as suggested by Corollary 3.1, this is also re\ufb02ected in\ntheir search performance. The MM-tree has comparable quantization performance to the 2M-tree\nand PA-tree. However, in the case of search, the MM-tree outperforms PA-tree in all datasets. This\ncan be attributed to the large margin partitions in the MM-tree. The comparison to 2M-tree is not\nas apparent. The MM-tree and PA-tree have \u03c9-balanced splits for small \u03c9 enforced algorithmically,\nresulting in bounded depth and bounded computation of O(l + n(1 + \u03c9)l/2l) for any given depth\nl. No such balance constraint is enforced in the 2-means algorithm, and hence, the 2M-tree can be\nheavily unbalanced. The absence of complete BSP 2M-tree beyond depth 4 and 6 in Figures 2 (b)\n& (c) respectively is evidence of the lack of balance in the 2M-tree. This implies possibly more\ncomputation and hence lower errors. Under these conditions, the MM-tree with an explicit balance\nconstraint performs comparably to the 2M-tree (slightly outperforming in 3 of the 4 cases) while\nstill maintaining a balanced tree (and hence returning smaller candidate sets on average).\n\n3A subset of a growth-restricted metric space (S, (cid:96)2) may not be growth-restricted. However, in our case,\nwe are not considering all subsets; we only consider subsets of the form (A \u2229 S) where A \u2282 RD is a convex\nset. So our condition does not imply that all subsets of (S, (cid:96)2) are growth-restricted.\n\n4We empirically explore the effect of the tree type on the violation of the inlier condition (C4) in Appendix\nB. The results imply that for any \ufb01xed value of \u03b7, almost the same number of alternate splits would be invoked\nfor the construction of different types of trees on the same dataset. Moreover, with \u03b7 \u2265 8, for only one of the\ndatasets would a signi\ufb01cant fraction of the partitions in the tree (of any type) need to be the alternate partition.\n\n6\n\n\f(a) OPTDIGITS\n\n(b) TINY IMAGES\n\n(c) MNIST\n\n(d) IMAGES\n\nFigure 2: Performance of BSP-trees with increasing traversal depth. The top row corresponds to quanti-\nzation performance of existing trees and the bottom row presents the nearest-neighbor error (in terms of mean\nrank \u03c4 of the candidate neighbors (CN)) of Algorithm 1 with these trees. The nearest-neighbor search error\ngraphs are also annotated with the mean distance-error of the CN (please view in color).\n4 Large margin BSP-tree\nWe established that the search error depends on the quantization performance and the partition mar-\ngins of the tree. The MM-tree explicitly maximizes the margin of every partition and empirical\nresults indicate that it has comparable performance to the 2M-tree and PA-tree in terms of the quan-\ntization performance. In this section, we establish a theoretical guarantee for the MM-tree quanti-\nzation performance. The large margin split in the MM-tree is obtained by performing max-margin\nclustering (MMC) with 2 clusters. The task of MMC is to \ufb01nd the optimal hyperplane (w\u2217, b\u2217) from\nthe following optimization problem5 given a set of points S = {x1, x2, . . . , xm} \u2282 RD:\n\nmin\nw,b,\u03bei\n\n(cid:107)w(cid:107)2\n\n2 + C\n\n1\n2\n\nm(cid:88)\n\ni=1\n\n\u03bei\n\ns.t.\n\n|(cid:104)w, xi(cid:105) + b| \u2265 1 \u2212 \u03bei, \u03bei \u2265 0 \u2200i = 1, . . . , m\n\nsgn((cid:104)w, xi(cid:105) + b) \u2264 \u03c9m.\n\n\u2212\u03c9m \u2264 m(cid:88)\n\ni=1\n\n(5)\n\n(6)\n\n(7)\n\n(8)\n\nMMC \ufb01nds a soft max-margin split in the data to obtain two clusters separated by a large (soft)\nmargin. The balance constraint (Equation (7)) avoids trivial solutions and enforces an \u03c9-balanced\nsplit. The margin constraints (Equation (6)) enforce a robust separation of the data. Given a solution\nto the MMC, we establish the following quantization error improvement rate for the MM-tree:\nTheorem 4.1. Given a set of points S \u2282 RD and a region A containing m points, consider an\n\u03c9-balanced max-margin split (w, b) of the region A into {Al, Ar} with at most \u03b1m support vectors\nand a split margin of size \u03b3 = 1/(cid:107)w(cid:107). Then the quantization error improvement is given by:\n\n\uf8eb\uf8ed1 \u2212 \u03b32 (1 \u2212 \u03b1)2(cid:16) 1\u2212\u03c9\n\n(cid:80)D\n\n1+\u03c9\n\n(cid:17)\n\n\uf8f6\uf8f8VS(A),\n\ni=1 \u03bbi\n\nVS({Al, Ar}) \u2264\n\nwhere \u03bb1, . . . , \u03bbD are the eigenvalues of the covariance matrix of A \u2229 S.\nThe result indicates that larger margin sizes (large \u03b3 values) and a smaller number of support vectors\n(small \u03b1) implies better quantization performance. Larger \u03c9 implies smaller improvement, but \u03c9 is\ngenerally restricted algorithmically in MMC. If \u03b3 = O(\n\u03bb1) then this rate matches the best possible\nquantization performance of the PA-tree (Table 1). We do assume that we have a feasible solution to\nthe MMC problem to prove this result. We use the following result to prove Theorem 4.1:\nProposition 4.1. [7, Lemma 15] Give a set S, for any partition {A1, A2} of a set A,\n\n\u221a\n\nVS(A) \u2212 VS({A1, A2}) =\n\n|A \u2229 S|2\nwhere \u00b5(A) is the centroid of the points in the region A.\n\n|A1 \u2229 S||A2 \u2229 S|\n\n(cid:107)\u00b5(A1) \u2212 \u00b5(A2)(cid:107)2 ,\n\n(9)\n\n5This is an equivalent formulation [16] to the original form of max-margin clustering proposed by Xu et al.\n(2005) [9]. The original formulation also contains the labels yis and optimizes over it. We consider this form\nof the problem since it makes our analysis easier to follow.\n\n7\n\n\fm(cid:88)\n\nThis result [7] implies that the improvement in the quantization error depends on the distance be-\ntween the centroids of the two regions in the partition.\nProof of Theorem 4.1. For a feasible solution (w, b, \u03bei|i=1,...,m) to the MMC problem,\n\nLet \u02dcxi = (cid:104)w, xi(cid:105)+b and mp = |{i : \u02dcxi > 0}| and mn = |{i : \u02dcxi \u2264 0}| and \u02dc\u00b5p = ((cid:80)\nand \u02dc\u00b5n = ((cid:80)\n\n|(cid:104)w, xi(cid:105) + b| \u2265 m \u2212 m(cid:88)\ni : \u02dcxi\u22640 \u02dcxi)/mn. Then mp \u02dc\u00b5p \u2212 mn \u02dc\u00b5n \u2265 m \u2212(cid:80)\n\nWithout loss of generality, we assume that mp \u2265 mn. Then the balance constraint (Equation (7))\ntells us that mp \u2264 m(1+ \u03c9)/2 and mn \u2265 m(1\u2212 \u03c9)/2. Then \u02dc\u00b5p\u2212 \u02dc\u00b5n + \u03c9(\u02dc\u00b5p + \u02dc\u00b5n) \u2265 2\u2212 2\ni \u03bei.\nSince \u02dc\u00b5p > 0 and \u00b5n \u2264 0, |\u02dc\u00b5p + \u02dc\u00b5n| \u2264 (\u02dc\u00b5p \u2212 \u02dc\u00b5n). Hence (1 + \u03c9)(\u02dc\u00b5p \u2212 \u02dc\u00b5n) \u2265 2 \u2212 2\ni \u03bei. For\nan unsupervised split, the data is always separable since there is no misclassi\ufb01cation. This implies\nthat \u03be\u2217\n\ni : \u02dcxi>0 \u02dcxi)/mp\n\ni \u2264 1\u2200i. Hence,\n\n(cid:80)\n\n(cid:80)\n\ni \u03bei.\n\n\u03bei.\n\ni=1\n\ni=1\n\nm\n\nm\n\n(cid:18)\n\n(cid:19)\n\n(cid:18) 1 \u2212 \u03b1\n\n(cid:19)\n\n1 + \u03c9\n\n\u02dc\u00b5p \u2212 \u02dc\u00b5n \u2265\n\n2 \u2212 2\nm\n\n|{i : \u03bei > 0}|\n\n/(1 + \u03c9) \u2265 2\n\n,\n\n(10)\n\n.\n\n1\n\ni=1\n\n\u03bbi.\n\n1 + \u03c9\n\n1 + \u03c9\n\n(cid:19)\n\nx\u2208A\u2229S\n\n(cid:19)2\n\n|A \u2229 S|\n\nVS(A) =\n\n= \u03b32 (1 \u2212 \u03b1)2\n\n(cid:18) 1 \u2212 \u03b1\n\n(cid:107)x \u2212 \u00b5(A)(cid:107)2 = tr (Cov(A \u2229 S)) =\n\nVS(A) \u2212 VS({Al, Ar}) \u2265 (1 \u2212 \u03c92)\u03b32\n\nsince the term |{i : \u03bei > 0}| corresponds to the number of support vectors in the solution.\nCauchy-Schwartz implies that (cid:107)\u00b5(Al) \u2212 \u00b5(Ar)(cid:107) \u2265 |(cid:104)w, \u00b5(Al) \u2212 \u00b5(Ar)(cid:105)|/(cid:107)w(cid:107) = (\u02dc\u00b5p \u2212 \u02dc\u00b5n)\u03b3,\nsince \u02dc\u00b5n = (cid:104)w, \u00b5(Al)(cid:105) + b and \u02dc\u00b5p = (cid:104)w, \u00b5(Ar)(cid:105) + b. From Equation (10), we can say\nthat (cid:107)\u00b5(Al) \u2212 \u00b5(Ar)(cid:107)2 \u2265 4\u03b32 (1 \u2212 \u03b1)2 / (1 + \u03c9)2. Also, for \u03c9-balanced splits, |Al||Ar| \u2265\n(1 \u2212 \u03c92)m2/4. Combining these into Equation (9) from Proposition 4.1, we have\n\n(11)\nLet Cov(A \u2229 S) be the covariance matrix of the data contained in region A and \u03bb1, . . . , \u03bbD be the\n(cid:88)\neigenvalues of Cov(A \u2229 S). Then, we have:\n\n(cid:18) 1 \u2212 \u03c9\nD(cid:88)\nThen dividing Equation (11) by VS(A) gives us the statement of the theorem.\n5 Conclusions and future directions\nOur results theoretically verify that BSP-trees with better vector quantization performance and large\npartition margins do have better search performance guarantees as one would expect. This means\nthat the best BSP-tree for search on a given dataset is the one with the best combination of good\nquantization performance (low \u03b2\u2217 in Corollary 3.1) and large partition margins (large \u03b3 in Corollary\n3.2). The MM-tree and the 2M-tree appear to have the best empirical performance in terms of the\nsearch error. This is because the 2M-tree explicitly minimizes \u03b2\u2217 while the MM-tree explicitly\nmaximizes \u03b3 (which also implies smaller \u03b2\u2217 by Theorem 4.1). Unlike the 2M-tree, the MM-tree\nexplicitly maintains an approximately balanced tree for better worst-case search time guarantees.\nHowever, the general dimensional large margin partitions in the MM-tree construction can be quite\nexpensive. But the idea of large margin partitions can be used to enhance any simpler space partition\nheuristic \u2013 for any chosen direction (such as along a coordinate axis or along the principal eigen-\nvector of the data covariance matrix), a one dimensional large margin split of the projections of the\npoints along the chosen direction can be obtained very ef\ufb01ciently for improved search performance.\nThis analysis of search could be useful beyond BSP-trees. Various heuristics have been developed\nto improve locality-sensitive hashing (LSH) [10]. The plain-vanilla LSH uses random linear projec-\ntions and random thresholds for the hash-table construction. The data can instead be projected along\nthe top few eigenvectors of the data covariance matrix. This was (empirically) improved upon by\nlearning an orthogonal rotation of the projected data to minimize the quantization error of each bin in\nthe hash-table [17]. A nonlinear hash function can be learned using a restricted Boltzmann machine\n[18]. If the similarity graph of the data is based on the Euclidean distance, spectral hashing [19]\nuses a subset of the eigenvectors of the similarity graph Laplacian. Semi-supervised hashing [20]\nincorporates given pairwise semantic similarity and dissimilarity constraints. The structural SVM\nframework has also been used to learn hash functions [21]. Similar to the choice of an appropriate\nBSP-tree for search, the best hashing scheme for any given dataset can be chosen by considering the\nquantization performance of the hash functions and the margins between the bins in the hash tables.\nWe plan to explore this intuition theoretically and empirically for LSH based search schemes.\n\n8\n\n\fReferences\n[1] J. H. Friedman, J. L. Bentley, and R. A. Finkel. An Algorithm for Finding Best Matches in\n\nLogarithmic Expected Time. ACM Transactions in Mathematical Software, 1977.\n\n[2] N. Verma, S. Kpotufe, and S. Dasgupta. Which Spatial Partition Trees are Adaptive to Intrinsic\nDimension? In Proceedings of the Conference on Uncertainty in Arti\ufb01cial Intelligence, 2009.\n[3] R.F. Sproull. Re\ufb01nements to Nearest-Neighbor Searching in k-dimensional Trees. Algorith-\n\nmica, 1991.\n\n[4] J. McNames. A Fast Nearest-Neighbor Algorithm based on a Principal Axis Search Tree. IEEE\n\nTransactions on Pattern Analysis and Machine Intelligence, 2001.\n\n[5] K. Fukunaga and P. M. Nagendra. A Branch-and-Bound Algorithm for Computing k-Nearest-\n\nNeighbors. IEEE Transactions on Computing, 1975.\n\n[6] D. Nister and H. Stewenius. Scalable Recognition with a Vocabulary Tree. In IEEE Conference\n\non Computer Vision and Pattern Recognition, 2006.\n\n[7] S. Dasgupta and Y. Freund. Random Projection trees and Low Dimensional Manifolds. In\n\nProceedings of ACM Symposium on Theory of Computing, 2008.\n\n[8] P. Ram, D. Lee, and A. G. Gray. Nearest-neighbor Search on a Time Budget via Max-Margin\n\nTrees. In SIAM International Conference on Data Mining, 2012.\n\n[9] L. Xu, J. Neufeld, B. Larson, and D. Schuurmans. Maximum Margin Clustering. Advances in\n\nNeural Information Processing Systems, 2005.\n\n[10] P. Indyk and R. Motwani. Approximate Nearest Neighbors: Towards Removing the Curse of\n\nDimensionality. In Proceedings of ACM Symposium on Theory of Computing, 1998.\n\n[11] T. Liu, A. W. Moore, A. G. Gray, and K. Yang. An Investigation of Practical Approximate\n\nNearest Neighbor Algorithms. Advances in Neural Information Proceedings Systems, 2005.\n\n[12] S. Dasgupta and K. Sinha. Randomized Partition Trees for Exact Nearest Neighbor Search. In\n\nProceedings of the Conference on Learning Theory, 2013.\n\n[13] J. He, S. Kumar and S. F. Chang. On the Dif\ufb01culty of Nearest Neighbor Search. In Proceedings\n\nof the International Conference on Machine Learning, 2012.\n\n[14] Y. Freund, S. Dasgupta, M. Kabra, and N. Verma. Learning the Structure of Manifolds using\n\nRandom Projections. Advances in Neural Information Processing Systems, 2007.\n\n[15] D. R. Karger and M. Ruhl. Finding Nearest Neighbors in Growth-Restricted Metrics.\n\nProceedings of ACM Symposium on Theory of Computing, 2002.\n\nIn\n\n[16] B. Zhao, F. Wang, and C. Zhang. Ef\ufb01cient Maximum Margin Clustering via Cutting Plane\n\nAlgorithm. In SIAM International Conference on Data Mining, 2008.\n\n[17] Y. Gong and S. Lazebnik. Iterative Quantization: A Procrustean Approach to Learning Binary\n\nCodes. In IEEE Conference on Computer Vision and Pattern Recognition, 2011.\n\n[18] R. Salakhutdinov and G. Hinton. Learning a Nonlinear Embedding by Preserving Class Neigh-\n\nbourhood Structure. In Arti\ufb01cial Intelligence and Statistics, 2007.\n\n[19] Y. Weiss, A. Torralba, and R. Fergus. Spectral Hashing. Advances of Neural Information\n\nProcessing Systems, 2008.\n\n[20] J. Wang, S. Kumar, and S. Chang. Semi-Supervised Hashing for Scalable Image Retrieval. In\n\nIEEE Conference on Computer Vision and Pattern Recognition, 2010.\n\n[21] M. Norouzi and D. J. Fleet. Minimal Loss Hashing for Compact Binary Codes. In Proceedings\n\nof the International Conference on Machine Learning, 2011.\n\n[22] S. Lloyd. Least Squares Quantization in PCM. IEEE Transactions on Information Theory,\n\n28(2):129\u2013137, 1982.\n\n9\n\n\f", "award": [], "sourceid": 390, "authors": [{"given_name": "Parikshit", "family_name": "Ram", "institution": "Georgia Tech"}, {"given_name": "Alexander", "family_name": "Gray", "institution": "Georgia Tech"}]}