{"title": "muSSP: Efficient Min-cost Flow Algorithm for Multi-object Tracking", "book": "Advances in Neural Information Processing Systems", "page_first": 425, "page_last": 434, "abstract": "Min-cost flow has been a widely used paradigm for solving data association problems in multi-object tracking (MOT). However, most existing methods of solving min-cost flow problems in MOT are either direct adoption or slight modifications of generic min-cost flow algorithms, yielding sub-optimal computation efficiency and holding the applications back from larger scale of problems. In this paper, by exploiting the special structures and properties of the graphs formulated in MOT problems, we develop an efficient min-cost flow algorithm, namely, minimum-update Successive Shortest Path (muSSP). muSSP is proved to provide exact optimal solution and we demonstrated its efficiency through 40 experiments on five MOT datasets with various object detection results and a number of graph designs. muSSP is always the most efficient in each experiment compared to the three peer solvers, improving the efficiency by 5 to 337 folds relative to the best competing algorithm and averagely 109 to 4089 folds to each of the three peer methods.", "full_text": "muSSP: Ef\ufb01cient Min-cost Flow Algorithm for\n\nMulti-object Tracking\n\nCongchao Wang, Yizhi Wang, Yinxue Wang, Chiung-Ting Wu, Guoqiang Yu*\n\nDepartment of Electrical and Computer Engineering, Virginia Tech\n{ccwang, yzwang, yxwang90, ctwu, yug}@vt.edu\n\nAbstract\n\nMin-cost \ufb02ow has been a widely used paradigm for solving data association prob-\nlems in multi-object tracking (MOT). However, most existing methods of solving\nmin-cost \ufb02ow problems in MOT are either direct adoption or slight modi\ufb01cations\nof generic min-cost \ufb02ow algorithms, yielding sub-optimal computation ef\ufb01ciency\nand holding the applications back from larger scale of problems. In this paper, by\nexploiting the special structures and properties of the graphs formulated in MOT\nproblems, we develop an ef\ufb01cient min-cost \ufb02ow algorithm, namely, minimum-\nupdate Successive Shortest Path (muSSP). muSSP is proved to provide exact\noptimal solution and we demonstrated its ef\ufb01ciency through 40 experiments on \ufb01ve\nMOT datasets with various object detection results and a number of graph designs.\nmuSSP is always the most ef\ufb01cient in each experiment compared to the three peer\nsolvers, improving the ef\ufb01ciency by 5 to 337 folds relative to the best competing\nalgorithm and averagely 109 to 4089 folds to each of the three peer methods.\n\n1\n\nIntroduction\n\nMulti-object tracking (MOT) is a fundamental task in computer vision and has a wide range of\napplications from traf\ufb01c surveillance, self-driving cars to cell/particle tracking in microscopy images\n[7, 20, 15]. In recent years, the min-cost \ufb02ow formulation of MOT has enjoyed popularity and served\nas a workhorse in addressing MOT problems [21, 16, 4, 14, 13]. On the one hand, it is a result of the\nsubstantial improvement of object detectors which enables the tracking-by-detection strategy [20, 13].\nOn the other hand, this formulation has great \ufb02exibility, for example, it can automatically determine\nthe number of trajectories and deal with missing or spurious detections [15, 4]. Since the min-cost\n\ufb02ow problems have been well studied and there exist polynomial-time algorithms [1], it was natural\nto directly apply the existing algorithms or modify them slightly. Indeed, Zhang et al. [21] used the\ncost-scaling approach and Pirsiavash et al. [16] proposed to adopt the successive shortest path (SSP)\napproach. These approaches can guarantee global optimality and are widely considered as the most\nef\ufb01cient solvers for generic min-cost \ufb02ow problems. However, their ef\ufb01ciency is suboptimal by a\nlarge margin for the MOT problems [14] as con\ufb01rmed in our experiments.\nWith the ever increasing number of objects and duration of tracking, more ef\ufb01cient algorithms for\nsolving MOT min-cost \ufb02ow problems are in urgent need. In this paper, we identify several important\nspecial structures and properties of the graph in the MOT min-cost \ufb02ow problem and show that they\ncan be used to design ef\ufb01cient algorithms, resulting in a dramatic reduction of computation time.\nSpeci\ufb01cally, the graph in a min-cost \ufb02ow problem for MOT has the following four specialties: (1)\nIt is a directed acyclic graph (DAG) with single source node s and single sink node t. (2) All arcs\u2019\ncapacities are one. (3) Each detection is represented with a pair of nodes, a pre-node and a post-node,\nwith a transition arc between them to incorporate detection con\ufb01dence. (4) For each detection, the\npre-node is linked from s through an inward arc and the post-node is linked to t through an outward\narc. Such arcs allow every detection to be the start or the end of a trajectory. An example of a typical\nmin-cost \ufb02ow graph generated from MOT problem is shown in Fig. 1(b).\n\n33rd Conference on Neural Information Processing Systems (NeurIPS 2019), Vancouver, Canada.\n\n\fFigure 1: (a) Objects detected in three frames. The \ufb01rst frame has two detections and misses one.\nLines between detections are possible ways of linking them. Each line is associated with a cost.\nDetections 1, 3 and 6 should be linked together as a single trajectory. (b) Typical min-cost \ufb02ow model\nfor MOT problem. Detection i is represented by a pair of nodes: a pre-node oi and a post-node hi.\nThe source node s is linked to all pre-nodes and all post-nodes are linked to the sink node t. These\nedges are shown in dashed lines. Edges between detections are shown in blue.\n\nTaking advantage of the specialties, we designed the minimum-update successive shortest path\n(muSSP) algorithm, an ef\ufb01cient solver for the min-cost \ufb02ow in MOT problem. While inspired by SSP\nalgorithm [1, 16], muSSP made fundamental changes to its framework by leveraging the observation\nthat, in SSP, most updates in building the shortest path tree through Dijkstra\u2019s algorithm were wasted\n(actually only the shortest s-t paths were useful for \ufb01nding the min-cost \ufb02ow). As detailed in the\nmethod section and Fig. 2 and 3, muSSP consists of four strategies which were guided by theoretical\nanalysis of the graph properties. muSSP follows the philosophy of minimally updating the shortest\npath tree, only when it is necessary to identify the shortest s-t path.\nMore speci\ufb01cally, \ufb01rstly, if the cost of linking two detections is too high, we can always have a better\nassociation where they are in different trajectories, so these arcs can be safely deleted (specialty 3\nand 4). Secondly, it can be proved from specialty 1 and 2 that once an arc connected to either s or\nt becomes non-empty, the \ufb02ow in it will always be 1. Pruning these arcs decreases the graph size\nand, more importantly, makes the following search of s-t shortest paths much more ef\ufb01cient. Thirdly,\nspecialty 1 and 4 enable us to simultaneously augment multiple shortest s-t paths. Lastly, since we\nonly have one source and all edges have unit capacity (specialty 1 and 2), the nodes to be reviewed\nfor \ufb01nding shortest s-t path can be quickly targeted and form a special tree structure. Dijkstra\u2019s\nalgorithm can be modi\ufb01ed to leverage this structure, which dramatically decreases the computation.\nIt appears that muSSP belongs to the solvers of a dynamic single-source shortest path (dSSSP)\nproblem [17, 10, 18]. dSSSP aims to ef\ufb01ciently retrieve the shortest path of each reachable node from\nthe single source node, after the original graph being modi\ufb01ed. dSSSP was directly applied to MOT\nby one of our peer methods, FollowMe [14]. However, our problem is different from the generic\ndSSSP problem because dSSSP tries to re-build the whole shortest path tree, while we care only the\nshortest s-t path. A large amount of computation is wasted on updating those unrelated vertices.\nmuSSP has the same theoretical worst-case complexity as SSP, but brings dramatic ef\ufb01ciency boost in\nreal applications. The effectiveness of muSSP is evidenced by forty experiments on \ufb01ve MOT datasets\ncombined with three widely used graph design methods [16, 14, 19]. Because min-cost \ufb02ow was also\nfrequently used as a sub-routine to approximate the quadratic programming formulation for MOT\nproblems [13, 4], our experiments include both these two kinds of scenarios. Compared with three\npeer algorithms, SSP, FollowMe [14], and a well-known implementation of cost-scaling algorithm\ncs2 [12], muSSP is always the most ef\ufb01cient in all experiments, with an ef\ufb01ciency improvement\nranging from 5 to 337 folds relative to the best competing algorithm. Regarding to each individual\nalgorithm, muSSP is averagely 4089 times faster than SSP, 1590 times faster than FollowMe, and\n109 times faster than cs2. These improvements are achieved without sacri\ufb01cing the space ef\ufb01ciency.\n\n2 Problem Formulation\n\nThe problem of associating detections from all frames (data association) can be formulated as a\nunit capacity min-cost \ufb02ow problem on a DAG [21, 2] as in Fig.1. In this paper, the term \"object\"\nrepresents a physical object existing over time (e.g. a person), and \"detection\" indicates a detected\nsnapshot of an object at some time point. An object corresponds to a series (trajectory) of detections.\n\n2\n\n7867864534532121ts\ud835\udc5c\ufffd\u210e\ufffdCostforinclusionCostforlinking258647\ud835\udc36\ud835\udc5c\ud835\udc60\ud835\udc61(3,6)Frame1Frame2Frame331CostforlinkingMissed(b)(a)\fWe denote the graph built in min-cost \ufb02ow formulation of MOT as G(V, E, C), with node set V , arc\nset E, and real-valued arc cost function C. The graph has one source s and one sink t. For each\ndetection i, we create a pre-node oi and a post-node hi, and three arcs, (oi, hi), (s, oi) and (hi, t).\nAny possible spatiotemporal transition of an object is corresponding to some pair of detections, i and\nj (i before j in time), and an arc (hi, oj) is created for it. The capacity of any arc is 1. Any directed\npath between node u and node v on graph G is denoted as \u03c0G(u, v). Under such construction, we can\ninterpret each s-t path \u03c0G(s, t) as an object trajectory candidate, linking a sequence of detections.\nThe problem is then turned into selecting a set of object trajectories from all the candidates. In the\nmin-cost \ufb02ow formulation, this is done by sending \ufb02ow from s to t, and the s-t paths eventually\nwith \ufb02ow inside are selected. Due to unit capacities and the total unimodularity of the problem, the\ncapacity constraints can be reduced to fuv \u2208 {0, 1},\u2200(u, v) \u2208 E. The set of arc \ufb02ows is denoted by\nf = {fuv|(u, v) \u2208 E}. We ask for in-out balance at any v \u2208 V \\ {s, t} (conservation constraints),\nso a \ufb02ow with total integer amount K can only be sent through K distinctive s-t paths, re\ufb02ecting the\nassumption of non-overlapping object trajectories.\nThe selection of s-t paths is guided by the design of arc cost C. MOT looks for object trajectories\nwith stronger evidences of detection, smaller change of a single object across time, and stronger\nevidences of initial and terminate points. Accordingly, the arc cost C(oi, hi) re\ufb02ects the reward of\nincluding the detection i. C(hi, oj) encodes the similarity between detection i and j. C(s, oi) and\nC(hi, t) represent respectively how likely the detection i is the initial point or the terminate point of a\n(u,v)\u2208E C(u, v)fuv is optimal\nuv in arc (u, v) \u2208 E). The costs can be negative, and therefore min-cost\nand denoted by f\u2217 (with f\u2217\n\ufb02ow formulation automatically leads to the optimal amount of \ufb02ow when minimizing Cf low(f ).\nThis min-cost \ufb02ow problem can also be formulated in an integer linear programming form:\n\ntrajectory. The \ufb02ow f with the minimum overall cost Cf low(f ) =(cid:80)\n\nmin\n\n(u,v)\u2208EC(u, v)fuv\n\n(1)\n\n(cid:80)\nv:(v,u)\u2208Efvu =(cid:80)\n\nf\n\nand (cid:80)\n\ns.t. fuv \u2208 {0, 1}, for all (u, v) \u2208 E\n\n(2)\n(3)\nwhich is guaranteed to have global optimal solution [1]. In the rest of this paper, the term \"graph\"\nand symbol G(V, E, C) all represent the graph built in min-cost \ufb02ow formulation of MOT problem,\nand all discussions are speci\ufb01cally for this family of graphs. Besides, the terms node and vertex, edge\nand arc, shortest path and least-cost path will be used interchangeably.\n\nv:(u,v)\u2208Efuv, for all u \u2208 V \\ {s, t},\n\n3 Method\n\nmuSSP uses four major strategies that take advantage of the special properties of our problem (Fig. 2).\nWe will detail each strategy after giving an overview of the framework. Proofs for lemmas/theorems\nas well as the time/space complexity analysis can be found in the supplementary.\n\n3.1 Overall framework\n\nWe \ufb01rst de\ufb01ne residual graphs used by min-cost \ufb02ow solvers before we give an overview of muSSP.\nDe\ufb01nition 1. The residual graph Gr(V, Er, Cr) of G(V, E, C) with respect to a \ufb02ow f is generated\nby replacing each arc (u, v) \u2208 E by two residual arcs (u, v) \u2208 Er and (v, u) \u2208 Er, where\n(u, v) \u2208 Er has cost Cr(u, v) = C(u, v) and residual capacity ruv = 1 \u2212 fuv, while (v, u) \u2208 Er\nhas cost Cr(v, u) = \u2212C(u, v) and rvu = fuv.\nGiven an input graph G(V, E, C), muSSP \ufb01rst removes unnecessary edges in the graph cleaning\nmodule. Then an initial shortest path tree (TSP) is obtained. If the stopping criteria is not met, muSSP\nsends a unit \ufb02ow from s to t along the shortest s-t path in the graph (AugmentFlow). The stopping\ncriteria is de\ufb01ned in the same ways as the SSP algorithm in [16]. The multi-path \ufb01nding module\ntries to \ufb01nd extra s-t paths on which we can send \ufb02ow without updating the residual graph and\nshortest path distances (FindMultiPath). If it succeeds, we can directly obtain the next s-t shortest\npath (ExtractPath). Otherwise, the residual graph is updated based on the s-t path found above\n(ResidualGraph). Edges in residual graph that are proved to be never involved in future shortest s-t\npaths are clipped (ClipPermanentEdge). The algorithm then updates the shortest path tree and a\nnew s-t path is extracted.\n\n3\n\n\fFigure 2: (a) muSSP algorithm. The four modules/strategies are shown with different shaded colors.\n(b) Flowchart of muSSP. Each module has the same color as (a). (c) Graph cleaning module for\ndummy edge removal. The blue arc between h1 and o5 are thicker as it has larger arc cost. If\nC(h1, o5) > C(s, o5) + C(h1, t), arc (h1, o5) will be removed.\n\nIn the initialization step, we use topological sorting to \ufb01nd shortest path (DAG-SP). As the graph\ncontains negative-cost edges, we convert the edge cost to reduced cost as C d(u, v) = C(u, v) +\nd(u) \u2212 d(v) (ConvertEdgeCost), where d(v) is the cost of the shortest path from source node s to v.\nThe shortest paths found with the reduced cost can be proved to be the same as before [1]. The new\nedge costs are non-negative and Dijkstra\u2019s algorithm can be applied.\n\n3.2 The independent \ufb02ipping lemma\n\nWe present a lemma that plays an important role in the muSSP algorithm. Since each time we push a\nunit \ufb02ow in a unit-capacity residual graph, the residual graph can be updated by \ufb02ipping the direction\nof arcs on the path though which we push \ufb02ow. The costs of those \ufb02ipped arcs have the same absolute\nvalues as the original ones but have the opposite signs. We denote the set of all simple paths in G\nfrom u to v as \u03a0G(u, v). \u03a0\u2217\nG(u, v) is the set of all simple least-cost paths in G from u to v. The\ncorresponding least cost is denoted as d\u2217\nu(v). Tree is a special type of graph. If tree T is rooted at v0,\n\u03c0T (v0, vi) is uniquely determined and we replace it with \u03c0T (vi). A tree T (V (cid:48), E(cid:48), C) is embedded\nin graph G(V, E, C) if V (cid:48) \u2286 V , E(cid:48) \u2286 E.\nDe\ufb01nition 2. A single-source shortest path tree (SP) is a tree T (V (cid:48), E(cid:48), C) embedded in a graph\nG(V, E, C), rooted at node s, such that \u03c0T (v) \u2208 \u03a0\u2217\nExamples of SP are shown in Fig. 3(a,c,e). Searching for the shortest s-t path can be done by\nmaintaining an SP and dynamically updating it when changes in the graph happen. To update the\nSP in a changed graph, we (1) identify the nodes whose lowest-cost paths in the SP no longer exist\nin the new graph; (2) update their least-cost paths from s in the new graph and rebuild SP.\nDe\ufb01nition 3. Given a tree T (V (cid:48), E(cid:48), C) rooted at v0, we de\ufb01ne the branch node and the set\nof descendant nodes for a node v \u2208 V (cid:48): branch(v) = u \u2208 V (cid:48) such that (v0, u) \u2208 \u03c0T (v);\ndescendants(v) = {u \u2208 V (cid:48)|\u2203vx \u2208 V (cid:48), (v, vx) \u2208 \u03c0T (u)}.\n\nG(s, v) for all v reachable from s in G.\n\n4\n\n(b)(c)ts\ud835\udc36(\ud835\udc60,\ud835\udc5c\ufffd)\ud835\udc36(\u210e\ufffd,\ud835\udc61)\ud835\udc36(\u210e\ufffd,\ud835\udc5c\ufffd)7867864534532121(a)FlowpushingandmultipathfindingResidualgraphupdating/clippingShortestpathupdatingGraphcleaningInitializationExtractpathGraphFlow\ud835\udc53Section3.3Section3.4Section3.6Section3.5Costdecreases?Multi-Path?YYNN\fFigure 3: (a) A single source shortest path tree (SP) shown with bold edges. The red path is the\nshortest s-t path. (b) The shortest path in (a) is \ufb02ipped (shown in green). (c) SP based on the updated\nresidual graph in (b). The red path is the new shortest path. The path through t makes the branch\ncontaining the shortest s-t path large and we have more nodes to update. (d) Updated residual graph\nwith permanent edges clipped (red crosses). (e) With clipping, the branch containing the shortest path\nis much smaller than (c). (f) Two branches in the SP, each containing a shortest s-t path (red and\npurple). Only the red one connects to t in SP as it has a lower cost. We can push \ufb02ows on both paths\nwithout updating residual graph (Theorem 3). Post-nodes h8 and h6 are independent while h8 and h7\nare not (Def. 4). (g) After removing permanent edges in the residual graph (b), we need to update the\ndistances from s to the red nodes, because the shortest paths from s to these nodes no longer exist\ndue to the \ufb02ipping operations. These nodes form a 0-tree. Note that edges in SP of (a) all have zero\ncosts. We initialize the distances from s to them using external edges (in blue) connecting to them.\nFor example, pre-node o5 is linked by arc (h1,o5) and (s,o5), so the initial distance is 2. Based on the\ninitial distance, only a subset of red nodes need to be pushed to the heap (Lemma 7) in Dijkstra\u2019s\nalgorithm. Once a node is popped from the heap, we can batch update its descendants. For example,\nif o5 is \ufb01rstly popped, all its descendants will be assigned the same distance as o5 (Theorem 4).\nTwo branches are shown in Fig. 3(f). For a node v in an SP, its branch node is the second node in\npath \u03c0T (v). Here we show that once the shortest s-t path is \ufb02ipped and the residual graph is updated,\nall nodes whose distances need to be updated share the same branch node.\nDe\ufb01nition 4. In an SP, two nodes u and v are independent, if and only if branch(u) (cid:54)= branch(v).\nLemma 1 (Independent \ufb02ipping). Given a residual graph Gr(Vr, Er, Cr) and its SP denoted as T\nr, for node v \u2208 Vr that is independent\nrooted at s, if we \ufb02ip all edges in path \u03c0T (t) and get new graph G(cid:48)\nwith node t in T , its least-cost path \u03c0T (v) is still valid in the new graph, i.e., \u03c0T (v) \u2208 \u03a0\u2217\nG(cid:48)\nLemma 1 shows that when we \ufb02ip the shortest path \u03c0 = {(s, v0), (v0, v1), . . . , (vx, t)}, only the\nnodes whose branch node is v0 need update. Reducing the size of descendants(v0) directly helps to\nimprove the ef\ufb01ciency. The \ufb01rst two edge clipping strategies we proposed are to ful\ufb01ll this task.\n\n(s, v).\n\nr\n\n3.3 Graph cleaning with dummy edge clipping\nGiven a min-cost \ufb02ow graph G(V, E, C) for MOT problem, an edge (u, v) \u2208 E connecting a post-\nnode to a pre-node is de\ufb01ned as a dummy edge if C(u, v) > C(s, v) + C(u, t) (Fig. 2(c)), In graph\ncleaning module, this kind of edges are removed to reduce the graph size and the computational cost\n(ClipDummyEdge). Linkage between dissimilar objects tends to be a dummy edge. Therefore, we\nhypothesize this linkage will not be included in a path, which is proved in the lemma below:\n\n5\n\n(a)(b)(c)(d)(e)WithoutclippingWithclipping7867864534532121ts786786453453211ts2Residualgraphupdating:permanentedgeclipping7867864534532121ts78678653532121ts44767643453211s58t827867864534532121tsBranch1Branch2Shortestpathinbranch1Shortestpathinbranch2(g)(f)76786453453211ts\ud835\udc63\ufffd8200000000000000000235343Multi-pathfindingShortestpathtreeupdating\fLemma 2. No dummy edges will appear in any optimal solution.\n\nFrom lemma 2, we can immediately get the following theorem.\nTheorem 1. Given a graph G(V, E, C) for MOT, removing all its dummy edges does not in\ufb02uence\nthe optimality of the \ufb01nal solution.\n\n3.4 Residual graph with permanent edge clipping\n\nAfter the step of updating the residual graph in each iteration, nodes tend to quickly accumulate to\nthe same branch especially when the graph is sparse (Fig. 3(a,b,c)). The main reason is that after\n\ufb02ipping paths, the least-cost path from s to some nodes goes through the sink node t. However, we\nwill prove that paths from t to other nodes will never be part of a valid s-t shortest path. Therefore, all\npaths that go through t will be replaced by other paths in future iterations. Thus, the updates of those\nnodes should not consider the arc originating from t. Here we theoretically prove our observation\nand propose a way to diminish the effects of node accumulation in branches (Fig. 3(d,e)). This is the\nkey step (ClipPermanentEdge) in our residual graph updating module.\nDe\ufb01nition 5. Given a residual graph Gr(Vr, Er, Cr) in MOT, we de\ufb01ne the set of permanent edges\nas {(u, v) \u2208 Er|v = s OR u = t}.\nNow we show the permanent edges are not necessary in \ufb01nding a shortest path.\nLemma 3. Any s-t path with permanent edge is not a simple path.\nLemma 4. Given a residual graph, we can always \ufb01nd a shortest s-t path which is also a simple path,\nunless t cannot be reached from s.\n\nThe above two lemmas imply that we can always \ufb01nd a shortest s-t path with no permanent edges.\nTheorem 2. Given a residual graph Gr(Vr, Er, Cr), removing all its permanent edges does not\nin\ufb02uence the optimality of the \ufb01nal solution.\n\nWhen the \ufb02ow amount is 0, we have no permanent edge. Each time we instantiate a shortest s-t\npath and \ufb02ip the arcs on it, we create two permanent edges. From the above theorem, these two\nedges can be safely removed from current residual graph (Fig. 3(d)). Other than s and t, the nodes\nrelated to permanent edges can also be removed from the graph, but its contribution to the ef\ufb01ciency\nimprovement is quite limited compared with edge clipping.\n\n3.5 Multi-path \ufb01nding and \ufb02ipping\nDenote the sequentially instantiated shortest paths as {\u03c01, \u03c02, . . . , \u03c0K}. We observe that the SP in\nthe ith iteration may contain not only \u03c0i, but also the following shortest paths \u03c0i+1, \u03c0i+2, . . ., with\nonly t missed in them (Fig.3(f)). Simultaneously instantiating and \ufb02ipping all these shortest paths\nwill save the computation on converting edge costs and decrease the number of duplicated updates,\nbecause some nodes may be updated several times if we instantiate the shortest paths one by one (Fig.\n3(f)).\nHere we analogize the idea of A* algorithm and propose an ef\ufb01cient way to check if current SP can\ninstantiate more than one shortest path. For each node v in graph G(V, E, C), we de\ufb01ne a distance\nfunction dt(v), whose value is C(v, t) if (v, t) \u2208 E and \u221e otherwise. Though dt is not an admissible\nsearch heuristic as commonly used in A* algorithm, we can still \ufb01nd the shortest s-t path based on it:\nLemma 5. Given a residual graph Gr(Vr, Er, Cr), its SP rooted at s, and the shortest distance\nfunction d from s to each node, we have d(t) = min(d(v) + dt(v)), v \u2208 Vr \\ {t}.\nTheorem 3. Given a residual graph Gr(Vr, Er, Cr), its SP denoted as T (V (cid:48)\nr, Cr) rooted\nat s and a sorted list of {d(v) + dt(v)} with v \u2208 Vr with ascending order, if the k nodes\n{v1, v2, . . . , vk} that occupy the top k locations of the list are mutually independent, the k paths\n{\u03c0T (v1), \u03c0T (v2), . . . , \u03c0T (vk)} can be simultaneously instantiated as k shortest paths.\nBased on theorem 3, we can ef\ufb01ciently check whether we can instantiate multiple paths from current\nSP. This is achieved by function FindMultiPath given the information of the branches to be updated\n(IdentifyNode4Update). An example of multi-path \ufb02ipping is shown in Fig. 3(f).\n\nr , E(cid:48)\n\n6\n\n\f3.6 Batch updating and heap shrinking for shortest path tree\n\nIn the shortest path tree updating module, we will discuss two approaches that reduce the running time\nof Dijkstra\u2019s algorithm (DijkstraWithBatchProc). The \ufb01rst is to simultaneously update the distances\nof multiple nodes. The second is to push less nodes into the heap used in Dijkstra\u2019s algorithm.\nIf a tree has zero cost for all edges, we call it a 0-tree (Fig. 3(g)). A 0-tree can emerge after edge\ncost conversion and \ufb02ipping. We can utilize 0-tree to reduce the computational cost of updating\ndistances of nodes in each iteration. After identifying the nodes that must be updated for \ufb01nding the\nnext shortest s-t path, we can ful\ufb01ll this task using Dijkstra\u2019s algorithm, as the edge costs have been\nconverted to non-negative values. Given G and its SP denoted as T rooted at s, after conversion, T\nbecomes not only an SP but also a 0-tree. This property can help to accelerate Dijkstra\u2019s algorithm.\nFor clarity and simplicity, we assume the updating happens with only one path \ufb02ipped as is shown in\nFig.3(d). The condition of updating nodes after simultaneously \ufb02ipping multiple paths is the same.\nSuppose shortest s-t path to be \ufb02ipped in current iteration is \u03c0 = {(s, v0), (v0, v1), . . . , (vx, t)}.\nSince the SP is a 0-tree now, after \ufb02ipping and permanent edge clipping, the nodes to update can be\ndivided into two sets: {t} and another 0-tree rooted at vx (Fig. 3(g)). As t can be ef\ufb01ciently updated\nby the sorted list discussed in Theorem 3, here we only show how to update nodes in the 0-tree rooted\nat vx.\nLemma 6. Given a 0-tree T0(V0, E0, Cr) embedded in residual graph Gr(Vr, Er, Cr), if v \u2208\ndescendants(v0), d\u2217\nLemma 6 shows that for a node in the 0-tree, the shortest distance from s to its descendants nodes\nis smaller than or equal to the distance from s to itself. Besides, Dijkstra\u2019s algorithm always \ufb01rst\ndeals with node with shorter distances from s. Combining these two facts leads to our batch updating\nstrategy below.\nTheorem 4. In Dijkstra\u2019s algorithm, if the distance from s to a node v in a 0-tree is permanently\nlabeled as d(v), d(v) is also the permanent label for the nodes in descendants(v) that haven\u2019t been\npermanently labeled.\n\nu(v0), \u2200u \u2208 Vr,\u2200v0 \u2208 V0.\n\nu(v) \u2264 d\u2217\n\nIt shows that we can permanently label a batch of nodes each time in Dijkstra\u2019s algorithm (Fig. 3(g)).\nDijkstra\u2019s algorithm is commonly implemented with heap. Decreasing the heap size saves the time\nconsumption of popping/pushing operation on it and increases ef\ufb01ciency. Inspired by the idea in [17]\nfor dSSSP, we found the heap size can also be shrunk based on Lemma 6.\nThe way we update the 0-tree rooted at vx with Dijkstra\u2019s algorithm is to relax all the edges that link\nnodes outside the 0-tree to nodes inside it. Then push their temporary distances to the heap. Thus the\ninitial size of the heap will be the node number of the tree. We divide the nodes in the 0-tree into two\nsets: P and Q. Set P contains nodes that cannot be correctly updated without checking the other\nnodes in the 0-tree and set Q is the complementary set of P . Only nodes in Q need to be inserted to\nthe heap. We propose an ef\ufb01cient way to estimate the set of P and only need to insert the remaining\nnodes in the 0-tree into the heap.\nLemma 7. In a 0-tree, nodes with larger temporary distance labels than their parent belong to set P .\n\nFrom Lemma 7, we can use a breadth-\ufb01rst search starting from root (vx) of the 0-tree, pushing only\nnodes whose temporary distance labels are not larger than their parents in the 0-tree (Fig. 3(g)). In\nthe best case, the heap size could be 0 and updating of the 0-tree takes linear-time.\n\n4 Experiments\n\nWe conducted three sets of experiments for the detailed analysis of ef\ufb01ciency improvement. First,\nmuSSP is used to directly solve min-cost \ufb02ow based data association problems in MOT. Second,\nwe test the ef\ufb01ciency of muSSP as a sub-routine to iteratively approximate the quadratic program-\nming formulation in MOT. Third, to further understand the improvement, we examine the relative\ncontributions of each key strategy we proposed, which can be found in the supplementary.\nWe compared muSSP with three popular methods, SSP, FollowMe [14], and cs2 [12]. Written in C\nlanguage, cs2 is an ef\ufb01cient implementation of cost-scaling algorithm, which is widely considered\nas the best solver for generic min-cost \ufb02ow problems and was used in [21, 16] for MOT problems.\n\n7\n\n\fDatasets\n\nTable 1: Ef\ufb01ciency comparison on MOT datasets (in seconds)\nKITTI(reglets)\nseq11\n\nKITTI(DPM)\nseq11\n\nseq00\n\nseq14\n\nseq10\n\nseq00\n\nseq10\n\n18.4(108)\n3.4(20)\n11.8(70)\n0.17(1)\n\n(a) graph design from [16]\nSSP\nFollowMe\ncs2\nmuSSP\n(b) graph design from [14]\nSSP\nFollowMe\ncs2\nmuSSP\n(c) graph design from [19]\n1.2h(24.5k)\nSSP\n917.9(5.1k)\nFollowMe\n24.6(137)\ncs2\n0.18(1)\nmuSSP\nDatasets\n\n20.9(116)\n3.2(18)\n13.6(76)\n0.18(1)\n\nseq04\n\n(a) graph design from [16]\n1.8h(4.9k)\nSSP\n2.4h(6.5k)\nFollowMe\n441.3(337)\ncs2\n1.31(1)\nmuSSP\n(b) graph design from [14]\nSSP\nFollowMe\ncs2\nmuSSP\n\n1.6h(4.3k)\n2.3h(6.2k)\n137.2(103)\n1.33(1)\n\n142.8(269)\n12.7(24)\n88.4(167)\n0.53(1)\n\n266.9(523)\n34.4(67)\n98.6(193)\n0.51(1)\n\n68.0(200)\n6.4(19)\n42.6(125)\n0.34(1)\n\n77.9(223)\n9.6(28)\n45.1(129)\n0.35(1)\n\n10.0h(46.2k)\n3.7h(16.9k)\n184.3(236)\n0.78(1)\n\n4.5h(26.0k)\n1.0h(5.5k)\n78.7(127)\n0.62(1)\n\nCVPR19\n\nseq07\n\nseq06\n\n370.9(976)\n801.1(2.1k)\n73.7(194)\n0.38(1)\n\n513.7(1.3k)\n803.3(2.0k)\n35.7(87)\n0.41(1)\n\n23.5(235)\n43.4(434)\n16.2(162)\n0.10(1)\n\n21.2(236)\n36.2(402)\n6.7(74)\n0.09(1)\n\n136.2(619)\n220.6(1.0k)\n34.5(157)\n0.22(1)\n\n85.7(204)\n21.2(50)\n39.4(94)\n0.42(1)\n\n38.5(167)\n62.1(270)\n16.0(70)\n0.23(1)\n\n183.9(400)\n53.1(115)\n63.1(137)\n0.46(1)\n\n173.0(455)\n26.2(69)\n46.1(121)\n0.38(1)\n\n137.3(490)\n36.8(131)\n57.8(206)\n0.28(1)\n\n120.8(448)\n12.0(44)\n41.1(152)\n0.27(1)\n\n96.1(291)\n12.0(36)\n39.8(121)\n0.33(1)\n\n5.2h(37.6k)\n0.9h(6.7k)\n69.7(139)\n0.50(1)\n\nseq08\n\n6.2(89)\n1.4(20)\n4.3(61)\n0.07(1)\n\n9.4(117)\n3.8(48)\n5.0(63)\n0.08(1)\n\n9.1(129)\n1.8(26)\n7.2(103)\n0.07(1)\n\n12.8(160)\n3.1(39)\n6.0(75)\n0.08(1)\n\n9.4(156)\n1.5(25)\n5.7(95)\n0.06(1)\n\n18.1(227)\n5.9(74)\n6.3(78)\n0.08(1)\n\n437.6(4.4k)\n198.4(2.0k)\n5.9(59)\n0.10(1)\n\n235.8(1.6k)\n224.5(1.5k)\n8.5(57)\n0.15(1)\n\nETHZ(DPM)\nseq04\n\nseq03\n\nseq14\n\n31.8(245)\n6.4(49)\n10.3(79)\n0.13(1)\n\n54.7(421)\n15.3(118)\n8.9(69)\n0.13(1)\n\n246.8(1.8k)\n242.2(1.7k)\n7.7(55)\n0.14(1)\n\n605.9(3.0k)\n649.1(3.2k)\n12.8(64)\n0.20(1)\n\nPTC\n\nMid\n\nHigh\n(d) probability principled\n379.2(134)\n0.4h(106)\n0.3h(430)\n1.2h(339)\n20.2(7)\n71.5(5)\n13.10(1)\n2.82(1)\n\nLow\n\n10.7(41)\n40.4(155)\n3.0(12)\n0.25(1)\n\nSSP and our muSSP were implemented in C++. To perform a fair comparison, FollowMe was\nre-implemented from their python package in C++. The implementation details can be found in the\nsupplementary.\n\n4.1 Solving the direct min-cost \ufb02ow model of data association problems in MOT\nTo represent the wide range of real world applications, we selected four public datasets including\nthree natural image MOT datasets (ETHZ (BAHNHOF and JELMOLI) [8], KITTI-Car [11], MOT\nCVPR 2019 Challenge[6]) and one particle tracking dataset (ISBI12 Particle Tracking Challenge\n(PTC) [5]). For natural image MOT, we designed the graphs using three methods [16, 14, 19]. The\ndifference among them is how to measure the similarity between detections and thus leads to different\narc cost functions in the graphs. As [19] is speci\ufb01cally designed for road scene, it was only applied to\nKITTI-Car dataset. For particle tracking, we used a probability principled way to design the graph,\nas detailed in supplementary.\nAll the experiments were based on the detection results either included in the datasets or provided by\nthe authors of [16, 14, 19]. The number of the detections varies from \u223c7k to \u223c200k. We performed\nexperiments with 39 combinations of detection results, graph design methods, and datasets. Summary\nof the detected objects, frames and the vertices and edges of the resultant graphs can be found in the\nsupplementary.\nTable 1 records the computation time for all experiments on a single core of 2.40GHz Xeon(R) CPU\nE5-2630. Each cell of the table represents the time consumed by a speci\ufb01c method under a graph\ndesign. The time was reported in seconds and sometimes in hours as denoted by the appending letter\n\u2019h\u2019. The numbers in the brackets indicate how many folds the method is slower than the most ef\ufb01cient\none. Bold font indicates the most ef\ufb01cient method. Overall, muSSP is always the fastest method\nin all 39 experiments. For majority of the experiments, muSSP achieved sub-second performance,\nwhile peer methods need up to several hours. Averagely muSSP is 4193 times faster than SSP, 1630\ntimes faster than FollowMe and 111 times faster than cs2. It can be seen from the table that SSP\nand FollowMe vary greatly with different graph settings from seconds to several hours. Interestingly,\nas an improved version of SSP, FollowMe usually performs better than SSP but when the object\nnumber is large (e.g., CVPR19), its performance drops quickly and is even worse than SSP. This\nis because FollowMe does not explicitly know that whether the paths containing \ufb02ipped edge(s) in\ncurrent residual graph are all valid in updating the shortest path tree (Lemma 1). To make sure not to\nmiss those valid ones, FollowMe inserts the nodes incident to the \ufb02ipped edges to the heap used in\nDijkstra\u2019s algorithm \ufb01rst. This operation will make the heap size huge when we have a large number\n\n8\n\n\fof paths \ufb02ipped. Our experiment on heap shrinking strategy in supplementary clearly shows this.\ncs2 performs relatively stable and outperforms SSP and FollowMe when the number of objects is\nlarge. It is likely that the local updating strategy of cs2 scales better with graph size than SSP and\nFollowMe. Note that the tracking results comparison is not listed since the solutions obtained by our\nproposed approach and baseline approaches are the same. This is also true for the following quadratic\nprogramming problem where the initialization and step function in Frank-Wolfe algorithm are the\nsame for each method.\n\n4.2 Solving the min-cost \ufb02ow approximation to the high-order modeling in MOT\n\nHigh-order relationships between detected objects have been incorporated for more accurate tracking,\nwhich, however, leads to NP-hard problems [4, 13, 3]. Existing methods approximate the solution\nwith the help of Frank-Wolfe algorithm or Lagrangian relaxation, where min-cost \ufb02ow solvers were\nfrequently used as a sub-routine. Here we test the ef\ufb01ciency of our muSSP in approximating the\nsolution of a quadratic programming problem formulated in [4] for the tracking problem. The\nquadratic objective function is provided by the authors in their software package derived from the\nsequence \u2019Time_13-59/View_002\u2019 of dataset PETS09 S1.L1 [9]. We \ufb01rstly relax its integer solution\nconstraint and then use Frank-Wolfe algorithm to iteratively solve this quadratic programming\nproblem. In each iteration, the problem is reduced to a min-cost \ufb02ow problem. The graph has 5866\nvertices and 36688 edges, and the algorithm runs totally 400 iterations. Results are shown in Table 2.\nmuSSP is 19 times faster than SSP, 27 times faster than FollowMe and 30 times faster than cs2.\n\nTable 2: Ef\ufb01ciency comparison of solving quadratic programming problem\n\nFollowMe\n557.0(27)\n\ncs2\n\n613.1(30)\n\nMethod\nPETS S1.L1-2\n\nSSP\n\n391.9(19)\n\nmuSSP\n20.3(1)\n\n5 Conclusion\n\nIn this paper, we proposed an ef\ufb01cient yet exact min-cost \ufb02ow solver muSSP for the data association\nin MOT problems, taking advantage of the specialties of the graphs built in MOT problems. muSSP is\napplicable not only to the direct min-cost \ufb02ow modeling but also to the min-cost \ufb02ow approximation\nto the high-order modeling in MOT. The ef\ufb01ciency was demonstrated on a wide range of public\ndatasets combined with various object detection results and graph designs. We expect this large\ndegree of ef\ufb01ciency improvement will save computational time for existing applications, enable\nengineers to tackle larger scale of problems, and inspire researchers to build more accurate modeling\nfor tracking, for instance, re\ufb01ning iteratively the graph designs based on the tracking results.\n\nReferences\n[1] Ravindra K Ahuja, Thomas L Magnanti, and James B Orlin. Network \ufb02ows: Theory, algorithms,\n\nand applications. 2008.\n\n[2] Jerome Berclaz, Francois Fleuret, Engin Turetken, and Pascal Fua. Multiple object tracking\nIEEE transactions on pattern analysis and machine\n\nusing k-shortest paths optimization.\nintelligence, 33(9):1806\u20131819, 2011.\n\n[3] Asad A Butt and Robert T Collins. Multi-target tracking by lagrangian relaxation to min-\ncost network \ufb02ow. In Proceedings of the IEEE Conference on Computer Vision and Pattern\nRecognition, pages 1846\u20131853, 2013.\n\n[4] Visesh Chari, Simon Lacoste-Julien, Ivan Laptev, and Josef Sivic. On pairwise costs for network\n\ufb02ow multi-object tracking. In Proceedings of the IEEE Conference on Computer Vision and\nPattern Recognition, pages 5537\u20135545, 2015.\n\n[5] Nicolas Chenouard, Ihor Smal, Fabrice De Chaumont, Martin Ma\u0161ka, Ivo F Sbalzarini, Yuanhao\nGong, Janick Cardinale, Craig Carthel, Stefano Coraluppi, Mark Winter, et al. Objective\ncomparison of particle tracking methods. Nature methods, 11(3):281, 2014.\n\n9\n\n\f[6] Patrick Dendorfer, Hamid Rezato\ufb01ghi, Anton Milan, Javen Shi, Daniel Cremers, Ian Reid,\nStefan Roth, Konrad Schindler, and Laura Leal-Taixe. Cvpr19 tracking and detection challenge:\nHow crowded can it get?, 2019.\n\n[7] Patrick Emami, Panos M Pardalos, Lily Elefteriadou, and Sanjay Ranka. Machine learning meth-\nods for solving assignment problems in multi-target tracking. arXiv preprint arXiv:1802.06897,\n2018.\n\n[8] A. Ess, B. Leibe, K. Schindler, , and L. van Gool. A mobile vision system for robust multi-\nperson tracking. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR\u201908).\nIEEE Press, June 2008.\n\n[9] James Ferryman and Ali Shahrokni. Pets2009: Dataset and challenge. In 2009 Twelfth IEEE\nInternational Workshop on Performance Evaluation of Tracking and Surveillance, pages 1\u20136.\nIEEE, 2009.\n\n[10] Daniele Frigioni, Alberto Marchetti-Spaccamela, and Umberto Nanni. Fully dynamic algorithms\n\nfor maintaining shortest paths trees. Journal of Algorithms, 34(2):251\u2013281, 2000.\n\n[11] Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are we ready for autonomous driving?\nthe kitti vision benchmark suite. In 2012 IEEE Conference on Computer Vision and Pattern\nRecognition, pages 3354\u20133361. IEEE, 2012.\n\n[12] Andrew V Goldberg. An ef\ufb01cient implementation of a scaling minimum-cost \ufb02ow algorithm.\n\nJournal of algorithms, 22(1):1\u201329, 1997.\n\n[13] Roberto Henschel, Laura Leal-Taix\u00e9, Daniel Cremers, and Bodo Rosenhahn. Fusion of head\nand full-body detectors for multi-object tracking. In Proceedings of the IEEE Conference on\nComputer Vision and Pattern Recognition Workshops, pages 1428\u20131437, 2018.\n\n[14] Philip Lenz, Andreas Geiger, and Raquel Urtasun. Followme: Ef\ufb01cient online min-cost \ufb02ow\ntracking with bounded memory and computation. In Proceedings of the IEEE International\nConference on Computer Vision, pages 4364\u20134372, 2015.\n\n[15] Wenhan Luo, Junliang Xing, Anton Milan, Xiaoqin Zhang, Wei Liu, Xiaowei Zhao, and Tae-\nKyun Kim. Multiple object tracking: A literature review. arXiv preprint arXiv:1409.7618,\n2014.\n\n[16] Hamed Pirsiavash, Deva Ramanan, and Charless C Fowlkes. Globally-optimal greedy algorithms\n\nfor tracking a variable number of objects. In CVPR 2011, pages 1201\u20131208. IEEE, 2011.\n\n[17] Ganesan Ramalingam and Thomas Reps. An incremental algorithm for a generalization of the\n\nshortest-path problem. Journal of Algorithms, 21(2):267\u2013305, 1996.\n\n[18] Liam Roditty and Uri Zwick. On dynamic shortest paths problems. In European Symposium on\n\nAlgorithms, pages 580\u2013591. Springer, 2004.\n\n[19] Sarthak Sharma, Junaid Ahmed Ansari, J Krishna Murthy, and K Madhava Krishna. Beyond\npixels: Leveraging geometry and shape cues for online multi-object tracking. In 2018 IEEE\nInternational Conference on Robotics and Automation (ICRA), pages 3508\u20133515. IEEE, 2018.\n\n[20] Yi Wu, Jongwoo Lim, and Ming-Hsuan Yang. Online object tracking: A benchmark. In\nProceedings of the IEEE conference on computer vision and pattern recognition, pages 2411\u2013\n2418, 2013.\n\n[21] Li Zhang, Yuan Li, and Ramakant Nevatia. Global data association for multi-object tracking\nusing network \ufb02ows. In 2008 IEEE Conference on Computer Vision and Pattern Recognition,\npages 1\u20138. IEEE, 2008.\n\n10\n\n\f", "award": [], "sourceid": 209, "authors": [{"given_name": "Congchao", "family_name": "Wang", "institution": "Virginia Polytechnic Institute and State University"}, {"given_name": "Yizhi", "family_name": "Wang", "institution": "Virginia Tech"}, {"given_name": "Yinxue", "family_name": "Wang", "institution": "Virginia Tech"}, {"given_name": "Chiung-Ting", "family_name": "Wu", "institution": "Virginia Tech"}, {"given_name": "Guoqiang", "family_name": "Yu", "institution": "Virginia Tech"}]}