{"title": "Maxing and Ranking with Few Assumptions", "book": "Advances in Neural Information Processing Systems", "page_first": 7060, "page_last": 7070, "abstract": "PAC maximum selection (maxing) and ranking of $n$ elements via random pairwise comparisons have diverse applications and have been studied under many models and assumptions. With just one simple natural assumption: strong stochastic transitivity, we show that maxing can be performed with linearly many comparisons yet ranking requires quadratically many. With no assumptions at all, we show that for the Borda-score metric, maximum selection can be performed with linearly many comparisons and ranking can be performed with $\\mathcal{O}(n\\log n)$ comparisons.", "full_text": "Maxing and Ranking with Few Assumptions\n\nMoein Falahatgar Yi Hao Alon Orlitsky Venkatadheeraj Pichapati Vaishakh Ravindrakumar\n\nUniversity of California, San Deigo\n\n{moein,yih179,alon,dheerajpv7,vaishakhr}@ucsd.edu\n\nAbstract\n\nPAC maximum selection (maxing) and ranking of n elements via random pairwise\ncomparisons have diverse applications and have been studied under many models\nand assumptions. With just one simple natural assumption: strong stochastic tran-\nsitivity, we show that maxing can be performed with linearly many comparisons\nyet ranking requires quadratically many. With no assumptions at all, we show that\nfor the Borda-score metric, maximum selection can be performed with linearly\n\nmany comparisons and ranking can be performed withO(n log n) comparisons.\n\nIntroduction\n\n1\n\n1.1 Motivation\nMaximum selection (maxing) and sorting using pairwise comparisons are among the most practical\nand fundamental algorithmic problems in computer science. As is well-known, maxing requires\n\nn\u2212 1 comparisons, while sorting takes \u21e5(n log n) comparisons.\n\nThe probabilistic version of this problem, where comparison outcomes are random, is of signi\ufb01cant\ntheoretical interest as well, and it too arises in many applications and diverse disciplines. In sports,\npairwise games with random outcomes are used to determine the best, or the order, of teams or\nplayers. Similarly Trueskill [1] matches video gamers to create their ranking. It is also used for a\nvariety of online applications such as to learn consumer preferences with the popular A/B tests, in\nrecommender systems [2], for ranking documents from user clickthrough data [3, 4], and more. The\npopular crowd sourcing website GIFGIF [5] shows how pairwise comparisons can help associate\nemotions with many animated GIF images. Visitors are presented with two images and asked to\nselect the one that better corresponds to a given emotion. For these reasons, and because of its\nintrinsic theoretical interest, the problem received a fair amount of attention.\n\n1.2 Terminology and previous results\nOne of the \ufb01rst studies in the area, [6] assumed n totally-ordered elements, where each comparison\n\u21b52 log 1\n\nerrs with the same, known, probability \u21b5< 1\n)\ncomparisons to output the maximum with probability\u2265 1\u2212 , and a ranking algorithm that uses\nO( n\n\n) comparisons to output the ranking with probability\u2265 1\u2212 .\n\n2. It presented a maxing algorithm that usesO( n\n\n\u21b52 log n\n\nThese results have been and continue to be of great interest. Yet this model has two shortcomings.\nIt assumes that there is only one random comparison probability, \u21b5, and that its value is known. In\npractice, comparisons have different, and arbitrary, probabilities, and they are not known in advance.\nTo address more realistic scenarios, researchers considered more general probabilistic models.\n\nConsider a set of n elements, without loss of generality[n] def= {1, 2, . . . , n}. A probabilistic model,\nor model for short, is an assignment of a preference probability pi,j \u2208 [0, 1] for every i \u2260 j \u2208\n[n], re\ufb02ecting the probability that i is preferred when compared with j. We assume that repeated\ncomparisons are independent and that there are no \u201cdraws\u201d, hence pj,i= 1\u2212 pi,j.\n\n31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA.\n\n\fIf pi,j\u2265 1\n2, we say that i is preferable to j and write i\u2265 j. Element i is maximal in a model if i\u2265 j\nfor all j \u2260 i. And a permutation `1, . . . ,` n is a ranking if `i \u2265 `j for all i\u2264 j. Observe that the\n\ufb01rst element of any ranking is always maximal. For example, for n= 3, p1,2= 1\uffff2, p1,3= 1\uffff3, and\np2,3= 2\uffff3, we have 1\u2265 2, 2\u2265 1, 3\u2265 1, and 2\u2265 3. Hence 2 is the unique maximum, and 2,3,1 is\n\nthe unique ranking. We seek algorithms that without knowing the underlying model, use pairwise\ncomparisons to \ufb01nd a maximal element and a ranking.\nTwo concerns spring to mind. First, there may be two elements i, j with pi,j arbitrarily close to\nhalf, requiring arbitrarily many comparisons just to determine which is preferable to the other. This\nconcern has a common remedy, that we also adopt. The PAC paradigm, e.g. [7, 8], that requires the\nalgorithm\u2019s output to be only Probably Approximately Correct.\n\nLet \u02dcpi,j\n\ndef= pi,j\u2212 1\n\n2.\n\n\u270f2 log n\n\n2, the more practical regime. For larger values of , one can use\n\nThe second concern is that not all models have a ranking, or even a maximal element. For example,\n\n. [12] derived a PAC maxing algorithm that usesO( n\n\n\u270f2 log n log n\n\nis preferable to the cyclically next, hence there is no maximal element and no ranking.\nA standard approach, that again we too will adopt, to address this concern is to consider structured\nmodels. The simplest may be parametric models, of which one of the more common is Placket\nLuce (PL) [10, 11], where each element i is associated with an unknown positive number ai and\n\n2 be the centered preference probability. Note that \u02dcpi,j \u2265 0 iff i is preferable to\nj. If \u02dcpi,j\u2265\u2212\u270f we say that i is \u270f-preferable to j. For 0< \u270f< 1\uffff2, an element i\u2208[n] is \u270f-maximum\n2 \u2265 > 0, a\nif it is \u270f-preferable to all other elements, namely, \u02dcpi,j \u2265 \u2212\u270f\u2200j \u2260 i. Given \u270f > 0, 1\nPAC maxing algorithm must output an \u270f-maxima with probability\u2265 1\u2212 , henceforth abbreviated\nwith high probability (WHP). Similarly, a permutation `1, . . . ,` n of{1, . . . ,n} is an \u270f-ranking if `i\nis \u270f-preferable to `j for all i\u2264 j, and a PAC ranking algorithm must output an \u270f-ranking WHP. Note\nthat in this paper, we consider \u2264 1\nour algorithms with = 1\nfor p1,2= p2,3= p3,1= 1, or the more opaque yet interesting non-transitive coins [9], each element\npi,j= ai\n\u270f) comparisons and a PAC\nai+aj\n\u270f) comparisons for any PL model. Related results for\nranking algorithm that usesO( n\nima and rankings. A model is strongly stochastically transitive (SST), if i\u2265 j and j \u2265 k imply\npi,k\u2265 max(pi,j, pj,k). By simple induction, every SST model has a maximum element and a rank-\ninequality if i\u2265 j and j\u2265 k imply that \u02dcpi,k\u2264 \u02dcpi,j+ \u02dcpj,k.\nO( n2\n) comparisons. For all models that satisfy both SST and triangle inequality, [7] derived\na PAC maxing algorithm that usesO( n\nshowed thatO\uffff n\n\uffff comparisons suf\ufb01ce and are optimal, and constructed a nearly-optimal\nPAC ranking algorithm that usesO( n log n(log log n)3\nofO((log log n)3) from optimum. Lower-bounds follow from an analogy to [15, 6]. Observe\n\nthe Mallows model under a non-PAC paradigm were derived by [13].\nBut signi\ufb01cantly more general, and more realistic, non-parametric, models may also have max-\n\ning. And one additional property, that is perhaps more dif\ufb01cult to justify, has proved helpful in\nconstructing maxing and sorting PAC algorithms. A tournament satis\ufb01es the stochastic triangle\n\nthat since the PL model satis\ufb01es both SST and triangle inequality, these results also improve the\ncorresponding PL results.\nFinally, we consider models that are not SST, or perhaps don\u2019t have maximal elements, rankings,\nor even their \u270f-equivalents. In all these cases, one can apply a weaker order relation. The Borda\n\nscore s(i) def= 1\nn\u2211j pi,j is the probability that i is preferable to another, randomly selected, element.\nElement i is Borda maximal if s(i)= maxj s(j), and \u270f-Borda maximal if s(i)\u2265 maxj s(j)\u2212 \u270f. A\nPAC Borda-maxing algorithm outputs an \u270f-Borda maximal element WHP (with probability\u2265 1\u2212 ).\nSimilarly, a Borda ranking is a permutation i1, . . . ,in such that for all 1\u2264 j\u2264 n\u2212 1, s(ij)\u2265 s(ij+1).\nAn \u270f-Borda ranking is a permutation where for all 1 \u2264 j \u2264 k \u2264 n, s(ij) \u2265 s(ik)\u2212 \u270f. A PAC\n\nBorda-ranking algorithm outputs an \u270f-Borda ranking WHP.\nRecall that Borda scores apply to all models. As noted in [16, 17, 8, 18] considering elements with\nnearly identical Borda scores shows that exact Borda-maxing and ranking requires arbitrarily many\ncomparisons.\n[8] derived a PAC Borda ranking, and therefore also maxing, algorithms that use\n\n\u270f) comparisons. [14] eliminated the log n\n\n) comparisons for all \u2265 1\n\n\u270f2\n\nIn Section 4 we show that if a model has a ranking, then an \u270f-ranking can be found WHP via\n\n\u270f factor and\n\nn, off by a factor\n\n\u270f2 log n\n\n\u270f2 log 1\n\n\u270f2 log n\n\n2\n\n\fO( n2\nlog( n\n)) PAC Borda ranking algorithm for restricted\n\u270f2) comparisons. [19] derived aO( n log n\nsetting. However note that several simple models, including p1,2= p2,3= p3,1= 1 do not belong to\n\nthis model.\n[20, 21, 22] considered deterministic adversarial versions of this problem that has applications\nin [23]. Finally, we note that all our algorithms are adaptive, where each comparison is cho-\nsen based on the outcome of previous comparisons. Non-adaptive algorithms were discussed\nin [24, 25, 26, 27].\n\n\u270f2\n\n2 Results and Outline\n\nOur goal is to \ufb01nd the minimal assumptions that enable ef\ufb01cient algorithms for these problems. In\nparticular, we would like to see if we can eliminate the somewhat less-natural triangle inequality.\nWith two algorithmic problems: maxing and ranking, and one property\u2013SST and one special metric\u2013\nBorda scores, the puzzle consists of four main questions.\n\nas with triangle inequality, and it matches the lower bound. 1b) No. In Section 4, Theorem 7, we\n\nWe essentially resolve all four questions. 1a) Yes. In Section 3, Theorem 6, we use SST alone to\n\n1) With just SST (and no triangle inequality) are there: a) PAC maxing algorithms withO(n) com-\nparisons? b) PAC ranking algorithms with nearO(n log n) comparisons? 2) With no assumptions\nat all, but for the Borda-score metric, are there: a) PAC Borda-maxing algorithms withO(n) com-\nparisons? b) PAC Borda-ranking algorithms with nearO(n log n) comparisons?\nderive aO\uffff n\n\uffff comparisons PAC maxing algorithm. Note that this is the same complexity\nshow that there are SST models where any PAC ranking algorithm with \u270f\u2264 1\uffff4 requires \u2326(n2)\ncomparisons. This is signi\ufb01cantly higher than the roughlyO(n log n) comparisons needed with\ntriangle inequality, and is close to theO(n2 log n) comparisons required without any assumptions.\nassumptions requiresO\uffff n\nO\uffff n\n\uffff comparisons.\n\n2a) Yes. In Section 5, Theorem 8, we derive a PAC Borda maxing algorithm that without any model\nIn Section 5,\nTheorem 9, we derive a PAC Borda ranking algorithm that without any model assumptions requires\n\n\uffff comparisons which is order optimal. 2b) Yes.\n\nBeyond the theoretical results sections, in Section 6, we provide experiments on simulated data. In\nSection 7, we discuss the results.\n\n\u270f2 log n\n\n\u270f2 log 1\n\n\u270f2 log 1\n\n3 Maxing\n\n3.1 SEQ-ELIMINATE\n\n2.\n\n\u270f2 log 1\n\n\u270f2 log n\n\nOur main building block is a simple, though sub-optimal, algorithm SEQ-ELIMINATE that sequen-\ntially eliminates one element from input set to \ufb01nd an \u270f-maximum under SST.\n\nBy SST, any element that is \u270f-preferable to absolute maximum element of S is an \u270f-maximum of\n\n\uffff comparisons and w.p.\u2265 1\u2212, \ufb01nds an \u270f-maximum. Even for sim-\nSEQ-ELIMINATE usesO\uffff n\npler models [15] we know that an algorithm needs \u2326\uffff n\n\uffff comparisons to \ufb01nd an \u270f-maximum\nw.p.\u2265 1\u2212 . Hence the number of comparisons used by SEQ-ELIMINATE is optimal up to a constant\nn but can be log n times the lower bound for = 1\nfactor when \u2264 1\nS. Therefore if we can reduce S to a subset S\u2032 of sizeO( n\nlog n) that contains an absolute maximum\n\uffff comparisons, we can then use SEQ-ELIMINATE to \ufb01nd an \u270f-maximum of\nof S usingO\uffff n\nS\u2032 and the number of comparisons is optimal up to constants. We provide one such reduction in\nelement in S if the latter is found to be better with con\ufb01dence\u2265 1\u2212 \uffffn. Note that if the running\nr with the competing element c if \u02dcpc,r \u2265 \u270f and retain r if \u02dcpc,r \u2264 0. If 0< \u02dcpc,r < \u270f, replacing or\n\nsubsection 3.2.\nSequential elimination techniques have been used before [13] to \ufb01nd an absolute maximum.\nIn\nsuch approaches, a running element is maintained, and is compared and replaced with a competing\n\nand competing elements are close to each other, this technique can take an arbitrarily long time to\ndeclare the winner. But since we are interested in \ufb01nding only an \u270f-maximum, SEQ-ELIMINATE\ncircumvents this issue. We later show that SEQ-ELIMINATE needs to update the running element\n\n\u270f2 log 1\n\n3\n\n\fretaining r doesn\u2019t affect the performance of SEQ-ELIMINATE signi\ufb01cantly. Thus, in other words\n\nAssuming that testing problem always returns the right answer, since SEQ-ELIMINATE never re-\n\nwe\u2019ve reduced the problem to testing whether \u02dcpc,r\u2264 0 or \u02dcpc,r\u2265 \u270f.\nplaces the running element with a worse element, either the output is the absolute maximum b\u2217 or b\u2217\nis never the running element. If b\u2217 is eliminated against running element r then \u02dcpb\u2217,r\u2264 \u270f and hence\n\nr is an \u270f-maximum and since the running element only gets better, the output is an \u270f-maximum.\nWe \ufb01rst present a testing procedure COMPARE that we use to update the running element in SEQ-\nELIMINATE.\n\n3.1.1 COMPARE\n\nof times i beats j, and let \u02c6\u02dcpi,j\n\nCOMPARE(i, j, \u270fl,\u270f u,) takes two elements i and j, and two biases \u270fu > \u270fl, and with con\ufb01dence\n\u2265 1\u2212 , determines whether \u02dcpi,j is\u2264 \u270fl or\u2265 \u270fu.\nFor this, COMPARE compares the two elements 2\uffff(\u270fu\u2212 \u270fl)2 log(2\uffff) times. Let \u02c6pi,j be the fraction\n2. If \u02c6\u02dcpi,j<(\u270fl+ \u270fu)\uffff2, COMPARE declares \u02dcpi,j\u2264 \u270fl (returns\n1), and otherwise it declares \u02dcpi,j\u2265 \u270fu (returns 2).\n\nDue to lack of space, we present the algorithm COMPARE in Appendix A.1 along with certain\nimprovements for better performance in practice .\nIn the Lemma below, we bound the number of comparisons used by COMPARE and prove its cor-\nrectness. Proof is in A.2.\n\ndef= \u02c6pi,j\u2212 1\n\nLemma 1. For \u270fu> \u270fl, COMPARE(i, j, \u270fl,\u270f u,) uses\u2264\n(\u270fu\u2212\u270fl)2 log 2\nthen w.p.\u2265 1\u2212 , it returns 1, else if \u02dcpi,j\u2265 \u270fu, w.p.\u2265 1\u2212 , it returns 2.\n\n2\n\n comparisons and if \u02dcpi,j\u2264 \u270fl,\n\nNow we present SEQ-ELIMINATE that uses the testing subroutine COMPARE and \ufb01nds an \u270f-\nmaximum.\n\n3.1.2 SEQ-ELIMINATE Algorithm\n\nSEQ-ELIMINATE takes a variable set S, selects a random running element r \u2208 S and repeatedly\nuses COMPARE(c, r, 0,\u270f, \uffffn) to compare r to a random competing element c\u2208 S\uffff r. If COMPARE\nreturns 1 i.e., deems \u02dcpc,r \u2264 0, it retains r as the running element and eliminates c from S, but if\nCOMPARE returns 2 i.e., deems \u02dcpc,r \u2265 \u270f, it eliminates r from S and updates c as the new running\n\nelement.\n\nAlgorithm 1 SEQ-ELIMINATE\n1: inputs\n2:\n\nSet S, bias \u270f, con\ufb01dence \n\n3: n\u2190\uffffS\uffff\n4: r\u2190 a random c\u2208 S, S= S\uffff{r}\n5: while S\u2260\uffff do\nPick a random c\u2208 S, S= S\uffff{c}.\nn)= 2 then\nif COMPARE(c, r, 0,\u270f, \nr\u2190 c\n\n6:\n7:\n8:\nend if\n9:\n10: end while\n11: return r\n\nness. Proof is in A.3.\n\nWe now bound the number of comparisons used by SEQ-ELIMINATE(S, \u270f, ) and prove its correct-\nTheorem 2. SEQ-ELIMINATE(S, \u270f, ) usesO\uffff\uffffS\uffff\u270f2 log\uffffS\uffff\uffff comparisons, and w.p.\u2265 1\u2212 outputs an\n\n\u270f-maximum.\n\n4\n\n\f3.2 Reduction\n\nelements in S.\n\n\u270f2 log 1\n\nn, here we present a\n\n3.2.1 Picking Anchor Element\n\nWe now present the subroutine PICK-ANCHOR that \ufb01nds a good anchor element.\n\nn, SEQ-ELIMINATE is order-wise optimal. For \u2265 1\n\nmaximum of S. Combining the reduction with SEQ-ELIMINATE results in an order-wise optimal\nalgorithm.\n\nRecall that, for \u2264 1\nreduction procedure that usesO\uffff n\n\uffff comparisons and w.p.\u2265 1\u2212 , outputs a subset S\u2032 of size\nO(\u221an log n) and an element a such that either a is a 2\u270f\uffff3-maximum or S\u2032 contains an absolute\nWe form the reduced subset S\u2032 by pruning S. We compare each element e \u2208 S with an anchor\nelement a, test whether \u02dcpe,a\u2264 0 or \u02dcpe,a\u2265 2\u270f\uffff3 using COMPARE, and retain all elements e for which\nCOMPARE returns the second hypothesis. For S\u2032 to be of sizeO(\u221an log n) we\u2019d like to pick an\nanchor element that is among the topO(\u221an log n) elements. But this can be computationally hard\nand we show that it suf\ufb01ces to pick an anchor that is not \u270f\uffff3-preferable to at mostO(\u221an log n)\nAn element a is called an(\u270f, n\u2032)-good anchor if a is not \u270f-preferable to at most n\u2032 elements, i.e.,\n\uffff{e\u2236 e\u2208 S and \u02dcpe,a> \u270f}\uffff\u2264 n\u2032.\nPICK-ANCHOR(S, n\u2032,\u270f, ) uses O\uffff n\nn\u2032\uffff comparisons and w.p.\u2265 1\u2212 , outputs an\n(\u270f, n\u2032)-good anchor element. PICK-ANCHOR \ufb01rst picks randomly a set Q of n\nfrom S without replacement. This ensures that w.p.\u2265 1\u2212 , Q contains at least one of the top n\u2032\nLet the absolute maximum element of Q be denoted as q\u2217. Now an \u270f-maximum of Q is \u270f-preferable\nto q\u2217. Further, if Q contains an element in the top n\u2032 elements, there exists n\u2212 n\u2032 elements worse\nthan q\u2217 in S. Thus by SST, the \u270f-maximum of Q is also \u270f-preferable to these n\u2212 n\u2032 elements and\nhence the output of PICK-ANCHOR is an(\u270f, n\u2032)-good anchor element. PICK-ANCHOR is shown in\nn\u2032\uffff comparisons and w.p.\u2265 1\u2212 ,\nLemma 3. PICK-ANCHOR(S, n\u2032,\u270f, ) usesO\uffff n\nn\u2032\u270f2 log 1\noutputs an(\u270f, n\u2032)-good anchor element.\n\uffff2\uffff comparisons where the con-\n\u270f2\ufffflog 1\nRemark 4. Note that PICK-ANCHOR(S, cn, \u270f, ) usesOc\uffff 1\n\nappendix A.4\nWe now bound the number of comparisons used by PICK-ANCHOR and prove its correctness. Proof\nis in A.5.\n\nstant depends only on c but not on n. Hence it is advantageous to use this method to pick near-\nmaximum element when n is large.\n\nelements. We then use SEQ-ELIMINATE to \ufb01nd an \u270f-maximum of Q.\n\n elements\n\nn\u2032 log 2\n\nn\u2032\u270f2 log 1\n\n log n\n\n log n\n\nWe now present PRUNE that takes an anchor element as input and prunes the set S using the anchor.\n\n3.2.2 Pruning\n\nPRUNE prunes S in multiple rounds. In each round t, for every element e in S, PRUNE tests whether\n\nGiven an(\u270fl, n\u2032)-good anchor element a, w.p.\u2265 1\u2212 \uffff2, PRUNE(S, a, n\u2032,\u270f l,\u270f u,) outputs a subset\nS\u2032 of size\u2264 2n\u2032. Further, any element e that is at least \u270fu-better than a i.e., \u02dcpe,a \u2265 \u270fu is in S\u2032\nw.p.\u2265 1\u2212 \uffff2.\n\u02dcpe,a\u2264 \u270fl or \u02dcpe,a\u2265 \u270fu using COMPARE(e, a, \u270fl,\u270f u,\uffff2t+1) and eliminates e if the \ufb01rst hypothesis i.e.,\n\u02dcpe,a\u2264 \u270fl is returned. By Lemma 1, an element e that is \u270fu better than a i.e., \u02dcpe,a\u2265 \u270fu passes the\ntth round of pruning w.p.\u2265 1\u2212 \uffff2t+1. Thus by union bound, the probability that such an element\nis not present in the pruned set is\u2264\u2211\u221et=1 \uffff2t+1\u2264 \uffff2.\nNow for element e that is not \u270fl-better than a i.e., \u02dcpe,a \u2264 \u270fl, by Lemma 1, the \ufb01rst hypothesis\nis returned w.p.\u2265 1\u2212 \uffff4. Hence w.h.p., the number of such elements (not \u270fl-better elements) is\nreduced by a factor of after each round. Since a is an(\u270fl, n\u2032)-good anchor element, there are at\nmost n\u2032 elements atleast \u270fl-better than a. Thus the number of elements left in the pruned set after\nround t is at most n\u2032+ nt. Thus PRUNE succeeds eventually in reducing the size to\u2264 2n\u2032 (in\n\u2264 log1\uffff\n\nn\u2032 rounds).\n\nn\n\n5\n\n\fAlgorithm 2 PRUNE\n1: inputs\n2:\n\nfor e in St do\n\n3: t\u2190 1\n4: S1\u2190 S\n5: while\uffffSt\uffff> 2n\u2032 and t< log2 n do\n\nSet S, element a, size n\u2032, lower bias \u270fl, upper bias \u270fu, con\ufb01dence .\nInitialize: Qt\u2190\uffff\nif COMPARE(e, a, \u270fl,\u270f u,\uffff2t+1)= 1 then\nQt\u2190 Qt\uffff{e}\nSt+1\u2190 St\uffff Qt\nt\u2190 t+ 1\n\n6:\n7:\n8:\n9:\n10:\n11:\n12:\n13:\n14: end while\n15: return St.\n\nend if\nend for\n\n2,\n\n\u270f2 log 1\n\n\u270f2 log 1\n\nn.\n\n3.3 Full Algorithm\n\nmaximum element of S.\n\nHere OPT-MAXIMIZE \ufb01rst \ufb01nds\n\n2, the output set contains an absolute\n\nWe now bound the number of comparisons used by PRUNE and prove its correctness. Proof is in\nA.6.\n\nn and a is an(\u270fl, n\u2032)-good anchor element, then w.p.\u2265 1\u2212 \n\uffff comparisons and outputs a set of size less than\nn(\u270fu\u2212\u270fl)2 log 1\n\nLemma 5. If n\u2032\u2265\u221a6n log n, \u2265 1\nPRUNE(S, a, n\u2032,\u270f l,\u270f u,) usesO\uffff\n2n\u2032. Further if a is not an \u270fu-maximum of S then w.p.\u2265 1\u2212 \nWe now present the main algorithm, OPT-MAXIMIZE that w.p.\u2265 1\u2212, usesO\uffff n\n\uffff comparisons\nand outputs an \u270f-maximum. For \u2264 1\nn, SEQ-ELIMINATE usesO( n\n) comparisons and hence\nwe directly use SEQ-ELIMINATE. Below we assume > 1\nan (\u270f\uffff3,\u221a6n log n)-good anchor\nPICK-ANCHOR(S,\u221a6n log n, \u270f\uffff3, \nThen using PRUNE(S, a,\u221a6n log n, \u270f\uffff3, 2\u270f\uffff3, \n4).\n4) with\na, OPT-MAXIMIZE prunes S to a subset S\u2032 of size \u2264 2\u221a6n log n such that if a is not a 2\u270f\uffff3\nmaximum i.e. \u02dcpb\u2217,a> 2\u270f\uffff3, S\u2032 contains the absolute maximum b\u2217 w.p.\u2265 1\u2212 \uffff2. OPT-MAXIMIZE\nthen checks if a is a 2\u270f\uffff3 maximum by using COMPARE(e, a, 2\u270f\uffff3,\u270f, \uffff(4n)) for every element\ne\u2208 S\u2032. If COMPARE returns \ufb01rst hypothesis for every e\u2208 S\u2032 then OPT-MAXIMIZE outputs a or else\nOPT-MAXIMIZE outputs SEQ-ELIMINATE(S\u2032,\u270f, \n4).\nNote that only one of these three cases is possible: (1) a is a 2\u270f\uffff3-maximum, (2) a is not an \u270f-\nmaximum and (3) a is an \u270f-maximum but not a 2\u270f\uffff3-maximum. In case (1), since a is a 2\u270f\uffff3-\nmaximum, by Lemma 1, w.p.\u2265 1\u2212 \uffff4, COMPARE returns the \ufb01rst hypothesis for every e\u2208 S\u2032 and\nOPT-MAXIMIZE outputs a. In both cases (2) and (3), as stated above, w.p.\u2265 1\u2212 \uffff2, S\u2032 contains\nthe absolute maximum b\u2217. Now in case (2) since a is not an \u270f-maximum, by Lemma 1, w.p.\u2265\n1\u2212 \uffff(4n), COMPARE(b\u2217, a, 2\u270f\uffff3,\u270f, \uffff(4n)) returns the second hypothesis. Thus OPT-MAXIMIZE\noutputs SEQ-ELIMINATE(S\u2032,\u270f, \uffff4), which w.p.\u2265 1\u2212 \uffff4, returns an \u270f-maximum of S\u2032 (recall that\nan \u270f-maximum of S\u2032 is an \u270f-maximum of S if S\u2032 contains b\u2217). Finally in case (3), OPT-MAXIMIZE\neither outputs a or SEQ-ELIMINATE(S\u2032,\u270f, \uffff4) and either output is an \u270f-maximum w.p.\u2265 1\u2212 .\n) comparisons and outputs an\nTheorem 6. W.p.\u2265 1\u2212 , OPT-MAXIMIZE(S, \u270f, ) usesO( n\n\nIn the below Theorem, we bound comparisons used by OPT-MAXIMIZE and prove its correctness.\nProof is in A.7.\n\nelement a using\n\n\u270f-maximum.\n\n\u270f2 log 1\n\n4 Ranking\nRecall that [14] considered a model with both SST and stochastic triangle inequality and derived\nn. By constrast, we consider a more\n\nan \u270f-ranking withO\uffff n log n(log log n)3\n\n\u270f2\n\n\uffff comparisons for = 1\n\n6\n\n\fAlgorithm 3 OPT-MAXIMIZE\n1: inputs\n2:\n\nSet S, bias \u270f, con\ufb01dence .\n\nn then\n\n4:\n5: end if\n\n3: if \u2264 1\nreturn SEQ-ELIMINATE(S, \u270f, )\n6: a\u2190 PICK-ANCHOR(S,\u221a6n log n, \u270f\uffff3, \n4)\n7: S\u2032\u2190 PRUNE(S, a,\u221a6n log n, \u270f\uffff3, 2\u270f\uffff3, \n4)\n8: for element e in S\u2032 do\nif COMPARE(e, a, 2\u270f\n4n)= 2 then\nreturn SEQ-ELIMINATE(S\u2032,\u270f, \n4)\n\n9:\n10:\nend if\n11:\n12: end for\n13: return a\n\n3 ,\u270f,\n\n\n\n8.\n\n\u270f2 log 1\n\n7\n\n2 and hence cannot achieve a con\ufb01dence of 7\n8.\n\nWe also present a trivial \u270f-ranking algorithm in Appendix B.2 that for any stochastic model with\n\nleast one comparison between a1 and an is less than 1\nProof sketch in B.1.\n\nthis model satis\ufb01es SST but not stochastic triangle inequality. Also note that any ranking where a1\n\nthe output of a comparison between any two elements other than a1 and an is essentially a fair coin\ntoss (since \u00b5 is very small). Thus if we output a ranking without querying comparison between a1\n2 since a1 and an must necessarily be ordered correctly.\n\ngeneral model without stochastic triangle inequality and show that even a 1\uffff4-ranking with just SST\ntakes \u2326(n2) comparisons for \u2264 1\nTo establish the lower bound, we reduce the problem of \ufb01nding 1\uffff4-ranking to \ufb01nding a coin with\nbias 1 among n(n\u22121)2\n\u2212 1 other fair coins. For this, we consider the following model with n elements\n2, \u02dcpai,aj = \u00b5(0< \u00b5< 1\uffffn10), when i< j and(i, j)\u2260(1, n). Note that\n{a1, a2, ..., an}: \u02dcpa1,an = 1\nprecedes an is an 1\uffff4-ranking and thus the algorithm only needs to order a1 and an correctly. Now\nand an, then the ranking is correct w.p.\u2248 1\nNow if an algorithm uses only n2\uffff20 comparisons then the probability that the algorithm queried at\nTheorem 7. There exists a model that satis\ufb01es SST for which any algorithm requires \u2326(n2) com-\nparisons to \ufb01nd a 1\uffff4-ranking with probability\u2265 7\uffff8.\nranking (Weak Stochastic Transitivity), usesO( n2\n) comparisons and outputs an \u270f-ranking\nw.p.\u2265 1\u2212 .\n) comparisons w.p.\u2265 1\u2212, we can \ufb01nd an \u270f-Borda\nWe show that for general models, usingO( n\nmaximum and usingO( n\nRecall that Borda score s(e) of an element e is the probability that e is preferable to an element\npicked randomly from S i.e., s(e) = 1\nn\u2211f\u2208S \u02dcpe,f . We \ufb01rst make a connection between Borda\nsetting, every arm a is associated with a parameter q(a) and pulling that arm results in a reward\nB(q(a)), a Bernoulli random variable with parameter q(a). Observe that we can simulate our\nrandom element where in our setting, for every element e, the associated parameter is s(e). Thus\ntoo. [28] and several others derived a PAC maximum arm selection algorithms that useO( n\n)\n) comparisons and w.p.\u2265 1\u2212 , outputs\nTheorem 8. There exists an algorithm that usesO( n\n\nPAC optimal algorithms derived under traditional bandit setting work for PAC Borda score setting\n\u270f2 log 1\ncomparisons and \ufb01nd an arm with parameter at most \u270f less than the highest. This implies an \u270f-Borda\nmaxing algorithm with the same complexity. Proof follows from reduction to Bernoulli multi-armed\nbandit setting.\n\n) comparisons w.p.\u2265 1\u2212 , we can \ufb01nd an \u270f-Borda ranking.\n\npairwise comparisons setting as a traditional bandit arms setting by comparing an element with a\n\nscores of elements and the traditional multi armed bandit setting.\n\nIn the Bernoulli multi armed\n\n5 Borda Scores\n\nan \u270f-Borda maximum.\n\n\u270f2 log n\n\n\u270f2 log n\n\n\u270f2 log 1\n\n\fFor \u270f-Borda ranking, we note that if we compare an element e with 2\n\n2. Ranking based on these approximate scores results in an \u270f-Borda ranking. We present BORDA-\n\u270f\nRANKING in C.1 that uses 2n\nin C.1.\n\n\u2265 1\u2212 \uffffn, the fraction of times e wins approximates the Borda score of e to an additive error of\n comparisons and w.p.\u2265 1\u2212 outputs an \u270f-Borda ranking. Proof\nTheorem 9. BORDA-RANKING(S, \u270f, ) uses 2n\n comparisons and w.p.\u2265 1\u2212 outputs an\n\n random elements, w.p.\n\n\u270f-Borda ranking.\n\n\u270f2 log 2n\n\n\u270f2 log 2n\n\n\u270f2 log 2n\n\n6 Experiments\n\nIn this section we validate the performance of our algorithms using simulated data. Since we es-\nsentially derived a negative result for \u270f-ranking, we consider only our \u270f-maxing algorithms - SEQ-\nELIMINATE and OPT-MAXIMIZE for experiments. All results are averaged over 100 runs.\n\n105\n\n16\n\n14\n\ny\nt\ni\n\n12\n\nOPT-MAXIMIZE\nSEQ-ELIMINATE\n\ny\nt\ni\n\nl\n\n \n\nx\ne\np\nm\no\nC\ne\np\nm\na\nS\n\nl\n\n10\n\n8\n\n6\n\n4\n\n2\n\n0\n\n0\n\n200\n\n400\n\n600\n\n800\n\n1200\nNumber of elements\n\n1000\n\n1400\n\n1600\n\n1800\n\n2000\n\n(a) small values of n\n\nl\n\nx\ne\np\nm\no\nC\ne\np\nm\na\nS\n\nl\n\n \n\n14\n\n12\n\n10\n\n8\n\n6\n\n4\n\n2\n\n0\n\n0\n\n106\n\nOPT-MAXIMIZE\nSEQ-ELIMINATE\n\n5000\n\n10000\n\n15000\n\nNumber of elements\n\n(b) large values of n\n\nFigure 1: Comparison of SEQ-ELIMINATE and OPT-MAXIMIZE\n\nder this model. In Figure 1, we compare the performance of SEQ-ELIMINATE and OPT-MAXIMIZE\n\nSimilar to [14, 7], we consider the stochastic model pi,j= 0.6\u2200i< j. We use maxing algorithms to\n\ufb01nd 0.05-maximum with error probability = 0.1. Note that i= 1 is the unique 0.05-maximum un-\nover different ranges of n. Figures 1(a), 1(b) show that for small n i.e., n\u2264 1300 SEQ-ELIMINATE\nperforms well and for large n i.e., n\u2265 1300, OPT-MAXIMIZE performs well. Since we are using\n= 0.1, the experiment suggests that for \uffff 1\nn1\uffff3 , OPT-MAXIMIZE uses fewer comparisons as com-\npared to SEQ-ELIMINATE. Hence it would be bene\ufb01cial to use SEQ-ELIMINATE for \u2264 1\nn1\uffff3 and\nOPT-MAXIMIZE for higher values of . In further experiments, we use = 0.1 and n< 1000 so we\n\nuse SEQ-ELIMINATE for better performance.\nWe compare SEQ-ELIMINATE with BTM-PAC [7], KNOCKOUT [14], MallowsMPI [13], and\nAR [16] . KNOCKOUT and BTM-PAC are PAC maxing algorithms for models with SST and\nstochastic triangle inequality requirements. AR \ufb01nds an element with maximum Borda score. Mal-\nlows \ufb01nds the absolute best element under Weak Stochastic Transitivity.\n\nWe again consider the model: pi,j= 0.6\u2200i< j and try to \ufb01nd a 0.05-maximum with error probability\n = 0.1. Note that this model satis\ufb01es both SST and stochastic triangle inequality and under this\n\nmodel all these algorithms can \ufb01nd an \u270f-maximum. From Figure 2(a), we can see that BTM-PAC\nperforms worse for even small values of n and from Figure 2(b), we can see that AR performs worse\nfor higher values of n. One possible reason is that BTM-PAC is tailored for reducing regret in the\nbandit setting and in the case of AR, Borda scores of elements become approximately the same with\nincreasing number of elements, leading to more comparisons. For this reason, we drop BTM-PAC\nand AR for further experiments.\n\nnumber of comparisons (57237) as SEQ-ELIMINATE (56683), PLPAC failed to \ufb01nd 0.09-maxima\n20 out of 100 runs whereas SEQ-ELIMINATE found the maximum in all 100 runs.\nIn \ufb01gure 3, we compare algorithms SEQ-ELIMINATE, KNOCKOUT [14] and MallowsMPI [13]\nfor models that do not satisfy stochastic triangle inequality. In Figure 3(a), we consider the stochastic\n\nWe also tried PLPAC [12] but it fails to achieve required accuracy of 1\u2212 since it is designed\nprimarily for Plackett-Luce. For example, we considered the previous setting pi,j= 0.6\u2200i< j with\nn= 100 and tried to \ufb01nd a 0.09-maximum with = 0.1. Even though PLPAC used almost same\nmodel p1,j = 1\n2+ \u02dcq\u22001< i< j where \u02dcq\u2264 0.05 and\nwe pick n= 10. Observe that this model satis\ufb01es SST but not stochastic triangle inequality. Here\n\n2+ \u02dcq\u2200j\u2264 n\uffff2, p1,j = 1\u2200j> n\uffff2 and pi,j = 1\n\n8\n\n\fy\nt\ni\n\nl\n\nx\ne\np\nm\no\nC\ne\np\nm\na\nS\n\nl\n\n \n\n106\n\n105\n\n104\n\n103\n\nSEQ-ELIMINATE\nKNOCKOUT\nMallowsMPI\nAR\nBTM-PAC\n\n7\n\n10\n\n15\n\nNumber of elements\n\n(a) small values of n\n\n109\n\n108\n\ny\nt\ni\n\nl\n\nx\ne\np\nm\no\nC\ne\np\nm\na\nS\n\nl\n\n \n\nSEQ-ELIMINATE\nKNOCKOUT\nMallowsMPI\nAR\n\n107\n\n106\n\n105\n\n104\n\n50\n\n100\n200\nNumber of elements\n\n500\n\n(b) large valuesof n\n\nFigure 2: Comparison of Maxing Algorithms with Stochastic Triangle Inequality\n\ntriangle inequality. We give an explanation for this behavior in Appendix D. By constrast, even\n\nFigure 3(a), we can see that MallowsMPI uses more comparisons as \u02dcq decreases since MallowsMPI\nis not a PAC algorithm and tries to \ufb01nd the absolute maximum. Even though KNOCKOUT performs\n\nagain, we try to \ufb01nd a 0.05-maximum with = 0.1. Note that any i\u2264 n\uffff2 is a 0.05 maximum. From\nbetter than MallowsMPI, it fails to output a 0.05 maximum with probability 0.12 for \u02dcq = 0.001\nand 0.26 for \u02dcq = 0.0001. Thus KNOCKOUT can fail when the model doesn\u2019t satisfy stochastic\nfor \u02dcq= 0.0001, SEQ-ELIMINATE outputted a 0.05 maximum in all runs and outputted the abosulte\nrameter . We consider n= 10 elements and \ufb01nd a 0.05-maximum with error probablility = 0.05.\n\nmaximum in 76% of trials. We can also see that SEQ-ELIMINATE uses much fewer comparisons\ncompared to the other two algorithms.\nIn Figure 3(b), we compare SEQ-ELIMINATE and MallowsMPI on the Mallows model, a model\nwhich doesn\u2019t satisfy stochastic triangle inequality. Mallows model can be speci\ufb01ed with one pa-\n\nFrom Figure 3(b) we can see that the performance of MallowsMPI gets worse as approaches 1,\nsince comparison probabilities get close to 1\n\n2 whereas SEQ-ELIMINATE is not affected.\n\nl\n\ny\nt\ni\nx\ne\np\nm\no\nC\ne\np\nm\na\nS\n\nl\n\n \n\n1012\n\n1010\n\n108\n\n106\n\n104\n\nSEQ-ELIMINATE\n\nKNOCKOUT\n\nMallowsMPI\n\n0.04\n\n0.02\n\n0.01\n\n0.001\n\n0.0001\n\nl\n\ny\nt\ni\nx\ne\np\nm\no\nC\ne\np\nm\na\nS\n\nl\n\n \n\n107\n\n106\n\n105\n\n104\n\n103\n\n102\n\n0\n\nSEQ-ELIMINATE\n\nMallowsMPI\n\n0.1\n\n0.2\n\n0.3\n\n0.4\n\n0.5\n\n0.6\n\n0.7\n\n0.8\n\n0.9\n\n1\n\n(a) No Triangle Inequality\n\n(b) Mallows Model\n\nFigure 3: Comparison of SEQ-ELIMINATE and MALLOWSMPI over Mallows Model\n\nOne more experiment is presented in Appendix E.\n\n7 Conclusion\n\nWe extended the study of PAC maxing and ranking to general models which satisfy SST but not\nstochastic triangle inequality. For PAC maxing, we derived an algorithm with linear complexity.\n\nFor PAC ranking, we showed a negative result that any algorithm needs \u2326(n2) comparisons. We\n\nthus showed that removal of stochastic triangle inequality constraint does not affect PAC maxing\nbut affects PAC ranking. We also ran experiments over simulated data and showed that our PAC\nmaximum selection algorithms are better than other maximum selection algorithms.\nFor unconstrained models, we derived algorithms for PAC Borda maxing and PAC Borda ranking\nby making connections with traditional multi-armed bandit setting.\n\n9\n\n\fAcknowledgments\nWe\n1619448.\n\nthank NSF for\n\nsupporting\n\nthis work\n\nthrough\n\ngrants CIF-1564355\n\nand CIF-\n\nReferences\n[1] Ralf Herbrich, Tom Minka, and Thore Graepel. Trueskill: a bayesian skill rating system. In Proceedings\nof the 19th International Conference on Neural Information Processing Systems, pages 569\u2013576. MIT\nPress, 2006. 1.1\n\n[2] Jialei Wang, Nathan Srebro, and James Evans. Active collaborative permutation learning. In Proceedings\nof the 20th ACM SIGKDD international conference on Knowledge discovery and data mining, pages\n502\u2013511. ACM, 2014. 1.1\n\n[3] Filip Radlinski and Thorsten Joachims. Active exploration for learning rankings from clickthrough data.\n\nIn Proceedings of the 13th ACM SIGKDD, pages 570\u2013579. ACM, 2007. 1.1\n\n[4] Filip Radlinski, Madhu Kurup, and Thorsten Joachims. How does clickthrough data re\ufb02ect retrieval\nquality? In Proceedings of the 17th ACM conference on Information and knowledge management, pages\n43\u201352. ACM, 2008. 1.1\n\n[5] http://www.gif.gf/. 1.1\n\n[6] Uriel Feige, Prabhakar Raghavan, David Peleg, and Eli Upfal. Computing with noisy information. SIAM\n\nJournal on Computing, 23(5):1001\u20131018, 1994. 1.2\n\n[7] Yisong Yue and Thorsten Joachims. Beat the mean bandit. In Proc. of the ICML, pages 241\u2013248, 2011.\n\n1.2, 6\n\n[8] R\u00b4obert Busa-Fekete, Bal\u00b4azs Sz\u00a8or\u00b4enyi, and Eyke H\u00a8ullermeier. Pac rank elicitation through adaptive sam-\n\npling of stochastic pairwise preferences. In AAAI, 2014. 1.2\n\n[9] Wikipedia. Nontransitive dice \u2014 Wikipedia, the free encyclopedia. http://en.wikipedia.org/w/\n[Online; accessed 19-May-\n\nindex.php?title=Nontransitive%20dice&oldid=779713322, 2017.\n2017]. 1.2\n\n[10] Robin L Plackett. The analysis of permutations. Applied Statistics, pages 193\u2013202, 1975. 1.2\n\n[11] R Duncan Luce. Individual choice behavior: A theoretical analysis. Courier Corporation, 2005. 1.2\n\n[12] Bal\u00b4azs Sz\u00a8or\u00b4enyi, R\u00b4obert Busa-Fekete, Adil Paul, and Eyke H\u00a8ullermeier. Online rank elicitation for\n\nplackett-luce: A dueling bandits approach. In NIPS, pages 604\u2013612, 2015. 1.2, 6\n\n[13] R\u00b4obert Busa-Fekete, Eyke H\u00a8ullermeier, and Bal\u00b4azs Sz\u00a8or\u00b4enyi. Preference-based rank elicitation using\n\nstatistical models: The case of mallows. In Proc. of the ICML, pages 1071\u20131079, 2014. 1.2, 3.1, 6, 6\n\n[14] Moein Falahatgar, Alon Orlitsky, Venkatadheeraj Pichapati, and Ananda Theertha Suresh. Maximum\nselection and ranking under noisy comparisons. In International Conference on Machine Learning, pages\n1088\u20131096, 2017. 1.2, 4, 6, 6, A.1, D\n\n[15] Yuan Zhou, Xi Chen, and Jian Li. Optimal pac multiple arm identi\ufb01cation with applications to crowd-\n\nsourcing. In International Conference on Machine Learning, pages 217\u2013225, 2014. 1.2, 3.1\n\n[16] Reinhard Heckel, Nihar B Shah, Kannan Ramchandran, and Martin J Wainwright. Active ranking from\npairwise comparisons and when parametric assumptions don\u2019t help. arXiv preprint arXiv:1606.08842,\n2016. 1.2, 6\n\n[17] Tanguy Urvoy, Fabrice Clerot, Raphael F\u00b4eraud, and Sami Naamane. Generic exploration and k-armed\n\nvoting bandits. In Proc. of the ICML, pages 91\u201399, 2013. 1.2\n\n[18] Kevin Jamieson, Sumeet Katariya, Atul Deshpande, and Robert Nowak. Sparse dueling bandits.\n\nArti\ufb01cial Intelligence and Statistics, pages 416\u2013424, 2015. 1.2\n\nIn\n\n[19] David Timothy Lee, Ashish Goel, Tanja Aitamurto, and Helene Landemore. Crowdsourcing for participa-\ntory democracies: Ef\ufb01cient elicitation of social choice functions. In Second AAAI Conference on Human\nComputation and Crowdsourcing, 2014. 1.2\n\n10\n\n\f[20] Jayadev Acharya, Ashkan Jafarpour, Alon Orlitsky, and Ananda Theertha Suresh. Sorting with adversarial\ncomparators and application to density estimation. In Information Theory (ISIT), 2014 IEEE International\nSymposium on, pages 1682\u20131686. IEEE, 2014. 1.2\n\n[21] Jayadev Acharya, Moein Falahatgar, Ashkan Jafarpour, Alon Orlitsky, and Ananda Theertha Suresh.\nMaximum selection and sorting with adversarial comparators and an application to density estimation.\narXiv preprint arXiv:1606.02786, 2016. 1.2\n\n[22] Mikl\u00b4os Ajtai, Vitaly Feldman, Avinatan Hassidim, and Jelani Nelson. Sorting and selection with imprecise\n\ncomparisons. ACM Transactions on Algorithms (TALG), 12(2):19, 2016. 1.2\n\n[23] Jayadev Acharya, Ashkan Jafarpour, Alon Orlitsky, and Ananda Theertha Suresh. Near-optimal-sample\n\nestimators for spherical gaussian mixtures. NIPS, 2014. 1.2\n\n[24] Arun Rajkumar and Shivani Agarwal. A statistical convergence perspective of algorithms for rank aggre-\n\ngation from pairwise data. In Proc. of the ICML, pages 118\u2013126, 2014. 1.2\n\n[25] Sahand Negahban, Sewoong Oh, and Devavrat Shah. Iterative ranking from pair-wise comparisons. In\n\nNIPS, pages 2474\u20132482, 2012. 1.2\n\n[26] Sahand Negahban, Sewoong Oh, and Devavrat Shah. Rank centrality: Ranking from pairwise compar-\n\nisons. Operations Research, 2016. 1.2\n\n[27] Minje Jang, Sunghyun Kim, Changho Suh, and Sewoong Oh. Top-k ranking from pairwise comparisons:\n\nWhen spectral ranking is optimal. arXiv preprint arXiv:1603.04153, 2016. 1.2\n\n[28] Yuan Zhou and Xi Chen. Optimal pac multiple arm identi\ufb01cation with applications to crowdsourcing.\n\n2014. 5\n\n11\n\n\f", "award": [], "sourceid": 3549, "authors": [{"given_name": "Moein", "family_name": "Falahatgar", "institution": "UCSD"}, {"given_name": "Yi", "family_name": "Hao", "institution": "UCSD"}, {"given_name": "Alon", "family_name": "Orlitsky", "institution": "University of California, San Diego"}, {"given_name": "Venkatadheeraj", "family_name": "Pichapati", "institution": "UC San Diego"}, {"given_name": "Vaishakh", "family_name": "Ravindrakumar", "institution": "UC San Diego"}]}