{"title": "Snap ML: A Hierarchical Framework for Machine Learning", "book": "Advances in Neural Information Processing Systems", "page_first": 252, "page_last": 262, "abstract": "We describe a new software framework for fast training of generalized linear models. The framework, named Snap Machine Learning (Snap ML), combines recent advances in machine learning systems and algorithms in a nested manner to reflect the hierarchical architecture of modern computing systems. We prove theoretically that such a hierarchical system can accelerate training in distributed environments where intra-node communication is cheaper than inter-node communication. Additionally, we provide a review of the implementation of Snap ML in terms of GPU acceleration, pipelining, communication patterns and software architecture, highlighting aspects that were critical for achieving high performance. We evaluate the performance of Snap ML in both single-node and multi-node environments, quantifying the benefit of the hierarchical scheme and the data streaming functionality, and comparing with other widely-used machine learning software frameworks. Finally, we present a logistic regression benchmark on the Criteo Terabyte Click Logs dataset and show that Snap ML achieves the same test loss an order of magnitude faster than any of the previously reported results, including those obtained using TensorFlow and scikit-learn.", "full_text": "Snap ML: A Hierarchical Framework for Machine\n\nLearning\n\nCelestine D\u00fcnner\u22171 Thomas Parnell\u22171\u2217\n\nDimitrios Sarigiannis1 Nikolas Ioannou1 Andreea Anghel1 Gummadi Ravi2\n\nMadhusudanan Kandasamy2 Haralampos Pozidis1\n\n1IBM Research, Zurich, Switzerland\n\n2IBM Systems, Bangalore, India\n\n{cdu,tpa,rig,nio,aan}@zurich.ibm.com\n{ravigumm,madhusudanan}@in.ibm.com\n\nhap@zurich.ibm.com\n\nAbstract\n\nWe describe a new software framework for fast training of generalized linear\nmodels. The framework, named Snap Machine Learning (Snap ML), combines\nrecent advances in machine learning systems and algorithms in a nested manner\nto re\ufb02ect the hierarchical architecture of modern computing systems. We prove\ntheoretically that such a hierarchical system can accelerate training in distributed\nenvironments where intra-node communication is cheaper than inter-node com-\nmunication. Additionally, we provide a review of the implementation of Snap ML\nin terms of GPU acceleration, pipelining, communication patterns and software\narchitecture, highlighting aspects that were critical for achieving high performance.\nWe evaluate the performance of Snap ML in both single-node and multi-node envi-\nronments, quantifying the bene\ufb01t of the hierarchical scheme and the data streaming\nfunctionality, and comparing with other widely-used machine learning software\nframeworks. Finally, we present a logistic regression benchmark on the Criteo\nTerabyte Click Logs dataset and show that Snap ML achieves the same test loss\nan order of magnitude faster than any of the previously reported results, including\nthose obtained using TensorFlow and scikit-learn.\n\n1\n\nIntroduction\n\nThe widespread adoption of machine learning and arti\ufb01cial intelligence has been, in part, driven by\nthe ever-increasing availability of data. Large datasets can enable training of more expressive models,\nthus leading to higher quality insights. However, when the size of such datasets grows to billions of\ntraining examples and/or features, the training of even relatively simple models becomes prohibitively\ntime consuming. Training can also become a bottleneck in real-time or close-to-real-time applications,\nin which one\u2019s ability to react to events as they happen and adapt models accordingly can be critical\neven when the data itself is relatively small.\nA growing number of small and medium enterprises rely on machine learning as part of their everyday\nbusiness. Such companies often lack the on-premises infrastructure required to perform the compute-\nintensive workloads that are characteristic of the \ufb01eld. As a result, they may turn to cloud providers in\norder to gain access to such resources. Since cloud resources are typically billed by the hour, the time\nrequired to train machine learning models is directly related to outgoing costs. For such an enterprise\ncloud user, the ability to train faster can have an immediate effect on their pro\ufb01t margin.\n\n\u2217equal contribution.\n\n32nd Conference on Neural Information Processing Systems (NeurIPS 2018), Montr\u00e9al, Canada.\n\n\fThe above examples illustrate the demand for fast, scalable, and resource-savvy machine learning\nframeworks. Today there is an abundance of general-purpose environments, offering a broad class of\nfunctions for machine learning model training, inference, and data manipulation. In the following\nwe will list some of the most prominent and broadly-used ones along with certain advantages and\nlimitations.\nscikit-learn [16] is an open-source module for machine learning in Python. It is widely used due to\nits user-friendly interface, comprehensive documentation and the wide range of functionality that it\noffers. While scikit-learn does not natively provide GPU support, it can call lower-level native C++\nlibraries such as LIBLINEAR to achieve high-performance. A key limitation of scikit-learn is that it\ndoes not scale to datasets that do not \ufb01t into the memory of a single machine.\nApache MLlib [10] is Apache Spark's scalable machine learning library. It provides distributed\ntraining of a variety of machine learning models and provides easy-to-use APIs in Java, Scala and\nPython. It does not natively support GPU acceleration, and while it can leverage underlying native\nlibraries such as BLAS, it tends to exhibit slower performance relative to the same distributed\nalgorithms implemented natively in C++ using high-performance computing frameworks such as\nMPI [6].\nTensorFlow [1] is an open-source software library for numerical computation using data \ufb02ow graphs.\nWhile TensorFlow can be used to implement algorithms at a lower-level as a series of mathematical\noperations, it also provides a number of high-level APIs that can be used to train generalized linear\nmodels (and deep neural networks) without needing to implement them oneself. It transparently\nsupports GPU acceleration, out-of-core operation, multi-threading and can scale across multiple nodes.\nWhen it comes to training of large-scale linear models, a downside of TensorFlow is the relatively\nlimited support for sparse data structures, which are frequently important in such applications.\nIn this work we will describe a new software framework for training generalized linear models\n(GLMs) that is designed to offer effective GPU-accelerated training in both single-node and multi-\nnode environments. In mathematical terms, the problems of interest can be expressed as the following\nconvex optimization problem:\n\n(cid:88)\n\ni\n\nmin\n\n\u03b1\n\nf (A\u03b1) +\n\ngi(\u03b1i)\n\n(1)\n\nwhere \u03b1 denotes the model to be learnt from the training data matrix A and f, gi are convex\nfunctions specifying the loss and regularization term. This general setup covers many primal and\ndual formulations of widely applied machine learning models such as logistic regression, support\nvector machines and sparse models such as lasso and elastic-net.\n\nContributions. The contributions of this work can be summarized as follows:\n\n\u2022 We propose a hierarchical version of the CoCoA framework [18] for training GLMs in distributed,\nheterogeneous environments. We derive convergence rates for such a scheme which show that a\nhierarchical communication pattern can accelerate training in distributed environments where\nintra-node communication is cheaper that inter-node communication.\n\n\u2022 We propose a novel pipeline for training on datasets that are too large to \ufb01t in GPU memory. The\npipeline is designed to maximize the utilization of the CPU, GPU and interconnect resources\nwhen performing out-of-core stochastic coordinate descent.\n\n\u2022 We review the implementation of the Snap ML framework, including its GPU-based local solver,\nstreaming CUDA operations, communication patterns and software architecture. We highlight\nthe aspects that are most critical in terms of performance, in the hope that some of these ideas may\nbe applicable to other machine learning software, including popular deep learning frameworks.\n\n2 System Overview\n\nWe start with a high-level, conceptual description of the Snap ML architecture. The core innovation\nof Snap ML is how multiple state-of-the-art algorithmic building blocks are nested to re\ufb02ect the\nhierarchical structure of a distributed systems. Our framework, as illustrated in Figure 1, implements\nthree hierarchical levels of data and compute parallelism in order to partition the workload among\ndifferent nodes in a cluster, taking full advantage of accelerator units and exploiting multi-core\nparallelism on the individual compute units.\n\n2\n\n\fLevel 1. The \ufb01rst level of parallelism spans across individual worker nodes in a cluster. The data is\ndistributed across the worker nodes that communicate via a network interface. This data-parallel\napproach serves to increase the overall memory capacity of our system and enables the training of\nlarge-scale datasets that exceed the memory capacity of a single machine.\nLevel 2. On the individual worker nodes we can leverage one or multiple GPU accelerators by\nsystematically splitting the workload between the host and the accelerator units. The different\nworkloads are then executed in parallel, enabling full utilization of the available hardware resources\non each worker node, thus achieving a second level of parallelism, across heterogeneous compute\nunits.\nLevel 3. To ef\ufb01ciently execute the workloads assigned to the individual compute units we leverage the\nparallelism offered by their respective compute architecture. We use specially-designed solvers to take\nfull advantage of the massively parallel architecture of modern GPUs and implement multi-threaded\ncode for processing the workload on CPUs. This results in an additional, third level of parallelism\nacross cores.\n\nFigure 1: Hierarchical structure of our distributed framework.\n\n2.1 Hierarchical Optimization Framework\n\nThe distributed algorithmic framework underlying Snap ML is a hierarchical version of the popular\nCoCoA method [18]. CoCoA is designed for communication-ef\ufb01cient distributed training of models\nof the form (1) across K worker nodes in a data-parallel setting. It assumes that the data matrix A\nis partitioned column-wise across the K workers and de\ufb01nes independent data-local optimization\ntasks for each of them. These tasks only require access to the local partition of the training data and\na shared vector v. This shared vector v is periodically exchanged over the network to synchronize\nthe work between the different nodes. Now, assume in addition to the K worker nodes we have\nL GPUs available on each node. A naive way to parallelize the work would be to setup CoCoA\nwith KL workers, de\ufb01ne KL local subproblems and assign one subproblem to each GPU. This\napproach, however, would require the synchronization of v between all GPUs in every round of the\nalgorithm and the performance would thus be limited be the slowest interconnect. To avoid this and\ntake advantage of the fast interconnect amongst the GPUs of the same node, we propose a nested\nversion of the CoCoA scheme which offers the possibility to perform multiple inner communication\nrounds within one outer communication round over the network. The local subproblems in the nested\nversion are de\ufb01ned according to [18] where we recursively apply the separable approximation to\nthe respective objectives. For an explicit statement of the local subproblems we refer the reader to\nthe Appendix. To give convergence guarantees for our hierarchical scheme we re\ufb01ne the existing\nconvergence results of CoCoA and combine these with tight convergence guarantees for the inner\nlevel of CoCoA derived by exploiting the speci\ufb01c structure of the subproblem objective.\nAssume the local subtasks are solved \u03b8-approximately according to the de\ufb01nition introduced in [18].\nThen, we can bound the suboptimality \u03b5 := F(\u03b1) \u2212 min\u03b1 F(\u03b1) after t1 iterations of CoCoA with\n\n3\n\n\ft2 inner iterations each as\n\n(cid:34)\n\nE[\u03b5] \u2264\n\n1 \u2212(cid:0)1 \u2212 (1 \u2212 \u03b8) 1\n\n4R2K\u03b2cA\n\nL\n\n(cid:35)\n\n(cid:1)t2\n\n1\nt1\n\n(2)\n\nwhere \u03b2 denotes the smoothness parameter of f, R is a bound on the support of gi and cA := (cid:107)A(cid:107)2.\nFor strongly convex gi a linear rate can be derived where we refer the reader to the appendix for\ndetailed proofs. For the special case where we choose to only do a single update in the inner level,\ni.e., t2 = 1, we recover the classical CoCoA scheme with KL workers.\nThe bene\ufb01t of the proposed hierarchical scheme becomes more signi\ufb01cant if the discrepancy between\nthe costs of intra-node and inter-node communication is large such as often found in modern cloud\ninfrastructures. Let us assume there is a cost c1 associated with communicating the shared vector v\nover the network and a cost c2 with communicating v between the GPUs within a node. Then, for a\ngiven cost budget C, the right-hand side of (2) can be optimized for t1, t2 to achieve the best accuracy\nunder a cost constraint C \u2264 t1t2ccomp + t1c1 + t1t2c2 where ccomp denotes the cost of computing a\n\u03b8-approximate solution on the subtasks.\n\n3\n\nImplementation Details\n\nIn this section we will describe implementation details of Snap ML starting with details of the\nGPU-based local solver and working up to the high-level APIs. We have attempted to highlight the\ncomponents that are most critical in terms of performance, in the hope that some of these ideas may\nbe applicable to other machine learning software, including popular deep learning frameworks.\n\n3.1 GPU Local Solver\n\nTo ef\ufb01ciently solve the optimization problem assigned to the GPU accelerators we implement the\ntwice-parallel asynchronous stochastic coordinate descent solver (TPA-SCD) [15, 14].\nExtension for Logistic Regression. In the previous literature [15, 14], TPA-SCD has been applied to\nridge regression, lasso and support vector machines. These objectives have the desirable property\nthat coordinate descent updates have closed-form solutions. In Snap ML, we also support objective\nfunctions for which this is not the case such as logistic regression. To address this issue, instead of\nsolving the coordinate-wise subproblem exactly, we make a single step of Newton\u2019s method, using\nthe previous value of the model as the initial point [22]. We \ufb01nd that the computations required to\ncompute the Newton step (i.e, the \ufb01rst and second derivative) can also be expressed in terms of a\nsimple inner product and thus the same TPA-SCD machinery can be applied.\nAdaptive Damping. A challenge arises when applying the asynchronous TPA-SCD algorithm to dense\ndatasets (or datasets which are globally sparse but locally dense in a few features) due to the fact\nthat a thread block on the GPU may have an inconsistent view of the shared vector v if it is reading\nwhile another thread block has only partially written its updates to the same vector in memory. These\ninconsistencies can lead to divergence in the coordinate descent algorithm. To alleviate this issue we\nhave implemented a damping heuristic, similar to that proposed in [23], to arti\ufb01cially slows down the\nmodel updates leading to more robust convergence behavior. We initialize the algorithm with the\ndamping parameter set to 1 (i.e., no damping) and after every sub-epoch on the GPU, verify that the\nvalue of the local subproblem has actually decreased. If it has not, we discard the current round of\nmodel updates, halve the value of the damping parameter and proceed. We note that the damping\nparameter may be adapted differently across data partitions. This adaptive scheme introduces the cost\nneeded to evaluate the value of the local subproblem within every sub-epoch, however this cost can\nbe mostly amortized into the TPA-SCD kernel and only requires an additional reduce operation, for\nwhich we use the DeviceReduce operator provided by the CUB library [17].\n\n3.2 Pipelining\n\nAsynchronous Data Streaming. When the data partition of each node is too large to \ufb01t into the\naggregate GPU memory on that node, we must employ out-of-core techniques to move the data\nin and out of GPU memory. One option is to split the data into batches and sequentially process\neach batch on the local GPUs. Snap ML also provides the ability to use DuHL [7] to dynamically\n\n4\n\n\fFigure 2: Data streaming pipeline.\n\nFigure 3: Example of snap-ml-mpi API.\n\ndetermine which set of data points that are most bene\ufb01cial to move into the GPU memory as the\ntraining progresses. Both of these schemes involve moving data over the CPU-GPU interconnect and\nwhile, for sparse models, it has been shown that when using DuHL the amount of data that needs to\nbe copied reduces with the number of rounds, there can still be some signi\ufb01cant overheads related to\ndata transfer for dense models. To alleviate these overheads we have developed an alternative, more\nhardware-optimized approach. We partition the GPU memory into an active buffer and a swap buffer.\nThen, using CUDA streams, we can perform TPA-SCD on the data in the active buffer while at the\nsame time copying the next batch of data into the swap buffer. As we shall show in Section 4, this\npipelined approach can allow one to completely hide the data transfer time behind the computation\ntime when using high-speed interconnects such as NVLINK.\nStreaming Permutation Generation. In order to implement TPA-SCD, we must generate a permutation\nof the coordinates in the active buffer. In order to fully utilize the available hardware, we introduce\na third pipeline stage to the training algorithm whereby the CPU is used to generate a set of 32-bit\npseudo-random numbers for the batch of data that is currently being transferred into the swap buffer.\nAt the start of the next round, we copy the random numbers onto the GPU device and use the\nDeviceRadixSort operator provided by CUB to sort the integers by index thus resulting in the required\npermutation in GPU memory. The sorting function templates provided by CUB have a signi\ufb01cant\nadvantage over those provided by the Thrust library [12] in that they properly support CUDA streams.\nThrust\u2019s sort by key internally allocates memory which is a blocking operation on the GPU, whereas\nCUB explicitly requires that all memory be allocated upfront. The resulting 3-stage pipeline is\nillustrated in Figure 2. In order to ensure that the pseudo-random number generator does not become\na bottleneck we implement a multi-threaded version of the highly ef\ufb01cient XORSHIFT algorithm [9].\n\n3.3 Communication Patterns\n\nIntra-node Communication. Within a single node, communication of the shared vector between the\nGPUs is handled within a single process using multi-threading. A thread is spawned to manage\neach GPU and within each local iteration, the updated shared vector is copied onto all devices using\nasynchronous CUDA memcpy operations. The GPU performs its updates and the changes to the\nshared vector are asynchronously copied back to the CPU and aggregated. After a number of local\niterations are completed, the local changes to the shared vector are aggregated over the network\ninterface. How exactly the updates to the shared vector are aggregated depends on whether the Spark\nor MPI API to Snap ML is used, as described in the next section.\nInter-node Communication. When using Spark, each node is managed by a Spark executor and the\nchanges to the global shared vector are copied over the JNI from the underlying shared library into\nthe JVM memory space and aggregated using Spark\u2019s reduce operator. The data is represented using\nraw byte arrays in order to minimize serialization/deserialization cost. The updated value of the\nshared vector is then communicated to each node using Spark\u2019s broadcast operator. When using MPI,\nan MPI process is spawned on each node and the global shared vector is updated in place using MPI\u2019s\nAllreduce operator.\nNUMA Locality. In order to achieve the maximal bandwidth provided by the CPU-GPU interconnect\nit is essential that the software framework is implemented in a NUMA-aware manner. Speci\ufb01cally, if\n\n5\n\n\fthe software is deployed on a two-socket node in which two GPUs are attached to each socket, it is\ncritical that the threads that manage data transfer to those GPUs be pinned to the correct socket. This\ncan be easily enforced using the functionality provided in the MPI rank\ufb01le that assigns the cores to\neach MPI process.\n\n3.4 Software Architecture\n\nC++ Template Library. The core functionality of Snap ML is implemented in C++/CUDA as a\nheader-only template library: libglm. It provides class templates for CPU, GPU and multi-GPU\nlocal solvers that can be instantiated with arbitrary data formats (e.g. sparse, dense, compressed) and\narbitrary objective functions mapping (1).\nLocal API. We provide a Python module, snap-ml-local, that adheres to the scikit-learn API and\ncan be used to accelerate training of GLMs in a non-distributed setting. This API is targeted at\nsingle-node users who wish to accelerate existing scikit-learn-based applications using one or more\nGPUs that are attached locally. This module exploits the functionality offered by libglm while being\n\ufb02exible in that it can be readily combined with additional functionality from scikit-learn such as data\nloading, pre-processing and evaluation metrics.\nMPI API. For users with larger data, who wish to perform training in a distributed environment we\nprovide an additional Python module: snap-ml-mpi. By importing this module, the users can describe\ntheir application using high-level Python code and then submit an MPI job on their cluster using\nmpirun specifying the nodes to be used for training. At run-time, the Python code makes calls to\nlibglm via an intermediate C++ layer that executes MPI operations to coordinate the training. The\nmodule also provides functions for ef\ufb01cient distributed data loading and evaluation of performance\nmetrics. An illustrative example is given in Figure 3.\nSpark API. Finally, for users who wish to perform distributed training on Apache Spark-managed\ninfrastructure we provide snap-ml-spark. This module is essentially a lightweight Py4J [5] wrapper\naround an underlying jar package that interacts with libglm via the Java Native Interface. Local data\npartitions are managed by libglm and reside in memory outside of the JVM, thus enabling ef\ufb01cient\nGPU acceleration. Apache Spark introduces a number of additional layers into the software stack and\nthus a number of associated overheads. For this reason, we typically observe that the performance of\nthe Spark-based deployments of Snap ML are slower than those using MPI [6].\n\n4 Experimental Results\n\nIn the following we will evaluate the performance of Snap ML and compare with some widely-used\nML frameworks in a single-node environment and a multi-node environment. Additionally, we will\nprovide an in-depth analysis of two key aspects: pipelining and hierarchical training.\nApplication and Datasets. We will focus on the application of click-through rate prediction (CTR),\nwhich is a binary classi\ufb01cation task. For our multi-node experiments we will use the Terabyte Click\nLogs dataset (criteo-tb) released by Criteo Labs [3]. It consists of 4.2 billion examples with 1 million\nfeature values. We use the data collected during the \ufb01rst 23 days for the training of our models\nand the last day for testing. The training data is 2.3TB in SVM Light format and is thus one of the\nlargest publicly available datasets, making it ideal for evaluating the performance of distributed ML\nframeworks. For our single-node experiments we use the smaller dataset released by Criteo Labs as\npart of their 2014 Kaggle competition (criteo-kaggle); the dataset has 45 million training examples\nand 1 million features. We perform a random 75%/25% train/test split. We obtained the preprocessed\ndata for both datasets from [2].\nInfrastructure. The results in this section were obtained using a cluster of 4 IBM Power Systems*\nAC922 servers. Each server has 4 NVIDIA Tesla V100 GPUs attached via the NVLINK 2.0 interface.\nThe nodes are connected via both an In\ufb01niBand network as well as a slower 1Gbit Ethernet interface.\nFor evaluation of the pipeline performance we also used an Intel x86-based machine (Xeon** Gold\n6150 CPU@2.70GHz ) with a single NVIDIA Tesla V100 GPU attached using the PCI Gen3 interface.\n\n6\n\n\fFigure 4: Single node benchmark (criteo-kaggle). Figure 5: Performance of hierachical CoCoA.\n\n4.1 Single-Node Performance\nWe benchmark the single node performance of Snap ML for the training of a logistic regression\nclassi\ufb01er against an equivalent solution in scikit-learn and TensorFlow. The same value of the\nregularization parameter was used in all cases. The different frameworks are used as follows:\nscikit-learn. We load a pickled version of the dataset and train a logistic regression classi\ufb01er in\nscikit-learn, with the option to solve the dual formulation enabled which allows faster training for this\napplication. Under the hood, scikit-learn is calling the LIBLINEAR library [8] to solve the resulting\noptimization problem. It operates in single-threaded mode and can not leverage any available GPU\nresources.\nTensorFlow. For the TensorFlow experiment, we \ufb01rst convert the svmlight data into the native binary\nformat for TensorFlow (TFRecord) using a custom parser. We then feed the TFRecord to a TensorFlow\nbinary classi\ufb01er (tf.contrib.learn.LinearClassi\ufb01er), treating the TFRecord features as sparse columns\nwith integerized features. We use the stochastic dual coordinated ascent optimizer provided by\nTensorFlow, using the optimizer and train input function options suggested by Google [11]. We use a\nbatch size of 1M, and a number of IO threads equal to the number of physical processors \u2013 settings\nwhich we have experimentally found to perform the best. The implementation is multi-threaded and\ncan leverage GPU resources (for the classi\ufb01er training and evaluation) if available. In this case we let\nTensorFlow use a single V100 GPU since we found it was faster than using all four.\nSnap ML. We load a pickled version of the dataset and train a logistic regression classi\ufb01er in Snap\nML using the snap-ml-local API. To compare with TensorFlow, we only allow Snap ML to use a\nsingle GPU. Since the criteo-kaggle dataset \ufb01ts into GPU memory, the streaming functionality of\nSnap ML is not active in this experiment.\nIn Figure 4, we compare the performance of the three aforementioned solutions. TensorFlow\nconverges in approximately 500 seconds whereas scikit-learn takes around 200 seconds. This\ndifference may be explained by the highly optimized C++ backend of scikit-learn for workloads that\n\ufb01t in memory, whereas TensorFlow processes data in batches 2. Finally, we can see that Snap ML\nconverges in around 20 seconds, an order of magnitude faster than the other frameworks.\n\n4.2 Out-of-core Performance\n\nIn order to evaluate the streaming performance of Snap ML we train a logistic regression model\nusing a single GPU for the \ufb01rst 200 million training examples of the criteo-tb dataset. We pro\ufb01le the\nexecution on a machine that uses the PCI Gen 3 interconnect and a machine that uses the NVLINK\n2.0 interconnect. In Figure 6a, we show the pro\ufb01ling results for the PCI-based setup. On stream\nS1, the random numbers for the next batch are copied (Init) and then the sorting and TPA-SCD are\nperformed (Train chunk) - this takes around 90ms. In stream S2 we copy the next data chunk onto\nthe GPU which takes around 318ms and is thus the bottleneck. In Figure 6b, for the NVLINK-based\nsetup we observe that the copy time is reduced to 55ms (almost a factor of 6), due to the faster\nbandwidth provided by NVLINK 2.0. This speed-up hides the data copy time behind the kernel\nexecution, effectively removing the copy time from the critical path and resulting in a 3.5x speed-up.\n\n2We did try to load the whole dataset in TensorFlow and not use batching, but there seems to be a known\n\nissue with TensorFlow for datasets that are bigger than 2GB [13].\n\n7\n\n\f(a) PCIe Gen 3 Interconnect.\n\n(b) NVLINK 2.0 Interconnect.\n\nFigure 6: Pipelined performance \u2018out-of-core\u2019.\n\n4.3 Hierarchical Scheme\n\nTo evaluate the effect of the hierarchical application of CoCoA, we train the \ufb01rst billion examples\nof the criteo-tb dataset using all 16 GPUs of the cluster. We train a logistic regression model from\nsnap-ml-mpi and evaluate the time to reach a target training suboptimality \u03b5 as a function of the\nnumber of inner CoCoA iterations performed (t2) using both a fast network (In\ufb01niBand) and a slow\nnetwork (1Gbit Ethernet). The scheme where t2 = 1 corresponds to the standard non-hierarchical\nCoCoA approach. The results, as plotted in Figure 5, show that there is only little bene\ufb01t to setting\nt2 > 1 when using the fast network since communication cost only accounts for a small fraction of\nthe overall training time, but when using the slow network it is possible to approach the fast network\nperformance by increasing t2. Such a scheme is therefore suitable for use in cloud-based deployments\nwhere high-performance networking is not normally available.\n\n4.4 Tera-Scale Benchmark\n\nTo evaluate the performance of Snap ML on criteo-tb, we use the snap-ml-mpi interface to train a\nlogistic regression classi\ufb01er using all 16 GPUs in the cluster. Because the data does not \ufb01t into the\naggregated memory of the GPUs the streaming functionality of Snap ML are active in this experiment.\nWe obtain a logarithmic loss on the test set of 0.1292 in 1.53 minutes. This is the total runtime\nincluding data loading, initialization and training time.\nThere have been a number of previously-published results on this same benchmark, using different\nmachine learning software frameworks, as well as different hardware resources. We will brie\ufb02y\nreview these results:\n\n\u2022 LIBLINEAR. In an experimental log posted in the libsvm datasets repository [2], the authors report\nusing LIBLINEAR-CDBLOCK [21] to perform training on a single machine with 128GB of RAM.\n\u2022 Vowpal Wabbit. In [19], the authors evaluated the performance of Vowpal Wabbit, a fast out-of-core\nlearning system. Training was performed on a 12 core (24 thread) machine with 128GB of memory\nusing Vowpal Wabbit 8.3.0 using the \ufb01rst 3 billion training examples of criteo-tb.\n\n\u2022 Spark MLlib. In the same benchmark [19], the authors also measured the performance of the\nlogistic regression provided by Spark MLlib. They deploy Spark 2.1.0 a cluster with total 512 cores\nand 2TB of memory. Each executor is giving 4 cores and 16TB of memory.\n\n\u2022 TensorFlow. Google have also published results where they use Google Cloud Platform to scale out\nthe training of a logistic regression classi\ufb01er from TensorFlow [20]. They report using 60 workers\nmachines and 29 parameter machines for the training of the full dataset.\n\n\u2022 TensorFlow on Spark. Criteo have published code [4] to train a logistic regression model that\nuses Tensor\ufb02ow together with Spark for distributing the training across multiples node. They also\nprovide results that were obtained using 12 Spark executors.\n\nIn Figure 7, we provide a visual summary of these results. We can observe that Snap ML on 16\nGPUs is capable of training such a model to a similar level of accuracy, 46x faster than the best\npreviously reported results, which was obtained using TensorFlow. In addition to the previously\n\n8\n\n\fFigure 7: Previously-published results for logistic regression on the Terabyte Click Logs dataset.\n\npublished results, we have also reproduced the TensorFlow results on our infrastructure, using the\noptimizer and train input function options suggested by Google [11] (similar to 4.1). We tuned the\nbatch size used by TensorFlow and found that Snap ML can train the logistic regression classi\ufb01er\nover 500\u00d7 faster than TensoFlow on exactly the same hardware.\n\n5 Conclusions\n\nIn this work we have described Snap ML, a new framework for fast training of generalized linear\nmodels. Snap ML can exploit modern computing infrastructure consisting of multiple machines\nthat contain both CPUs and GPUs. The framework is hierarchical in nature, allowing it to adapt to\ncloud-based deployments where the cost of communication between nodes may be relatively high. It\nis also able to effectively leverage modern high-speed interconnects to hide the cost of transferring\ndata between CPU and GPU when training on datasets that are too large to \ufb01t into GPU memory. We\nhave shown that Snap ML can provide signi\ufb01cantly faster training than existing frameworks in both\nsingle-node and multi-node benchmarks. On one of the largest publicly available datasets, we have\nshown that Snap ML can be used to train a logistic regression classi\ufb01er in 1.5 minutes: more than an\norder of magnitude faster than any of the previously reported results.\n\nAcknowledgement\n\nThe authors would like to thank Martin Jaggi for valuable input regarding the algorithmic structure\nof our system, Michael Kaufmann and Adrian Sch\u00fcpbach for testing and bug \ufb01xes, Kubilay Atasu\nfor contributing code for load balancing, and Manolis Sifalakis and Urs Egger for setting up vital\ninfrastructure. We would also like to thank our colleagues Christoph Hagleitner and Cristiano Malossi\nfor providing access to heterogeneous compute resources and providing valuable support when\nscheduling large-scale jobs. Finally, we would also like to thank Hillery Hunter, Paul Crumley and I-\nHsin Chung for providing access to the servers that were used to perform the tera-scale benchmarking,\nand Bill Armstrong for his guidance and support of this project.\n\n*Trademark, service mark, registered trademark of International Business Machines Corporation in the United\nStates, other countries, or both.\n** Intel Xeon is a trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States\nand other countries. Java and all Java-based trademarks and logos are trademarks or registered trademarks of\nOracle and/or its af\ufb01liates. TensorFlow, the TensorFlow logo and any related marks are trademarks of Google\nInc. The Apache Software Foundation (ASF) owns all Apache-related trademarks, service marks, and graphic\nlogos on behalf of our Apache project communities, and the names of all Apache projects are trademarks of the\nASF.\n\n9\n\nLIBLINEAR[1 core]Vowpal Wabbit[12 cores]Spark Mllib[512 cores]TensorFlow[60 worker machines, 29 parameter machines]Snap ML[16 V100 GPUs]TensorFlow[16 V100 GPUs]TensorFlow on Spark[12 executors]0.1280.1290.1300.1310.1320.133110100100010000LogLoss (Test)Training Time (minutes)\fReferences\n[1] Mart\u00edn Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado,\nAndy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey\nIrving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg,\nDandelion Man\u00e9, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens,\nBenoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda\nVi\u00e9gas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng.\nTensorFlow: Large-scale machine learning on heterogeneous systems, 2015. Software available from\nhttps://www.tensorflow.org/.\n\n[2] Chih-Chung Chang and Chih-Jen Lin. LIBSVM : a library for support vector machines. ACM Transactions\non Intelligent Systems and Technology, 2011. Software available at http://www.csie.ntu.edu.tw/\n~cjlin/libsvm.\n\n[3] Criteo Labs.\n\ndemic\ncriteo-releases-industrys-largest-ever-dataset/.\n\ncommunity,\n\nCriteo releases industry\u2019s largest-ever dataset\n\nfor machine learning to aca-\nhttps://www.criteo.com/news/press-releases/2015/07/\n\n2015.\n\n[4] Criteo Labs. Learning Click-Through Rate at Scale with Tensor\ufb02ow on Spark, 2018. https://github.\n\ncom/criteo/CriteoDisplayCTR-TFOnSpark.\n\n[5] Barthelemy Dagenais. Py4j: A bridge between python and java, 2018. Software available at https:\n\n//www.py4j.org.\n\n[6] Celestine D\u00fcnner, Thomas Parnell, Kubilay Atasu, Manolis Sifalakis, and Haris Pozidis. Understanding\nIn Proceedings of the IEEE\n\noptimizing distributed machine learning applications on apache spark.\nInternational Conference on Big Data, IEEEBigData\u201917, pages 99\u2013100, Boston, MA, December 2017.\n\n[7] Celestine D\u00fcnner, Thomas Parnell, and Martin Jaggi. Ef\ufb01cient use of limited memory accelerators for\nlinear learning on heterogeneous systems. In Advances in Neural Information Processing Systems 30,\nNIPS\u201917, pages 4261\u20134270, Long Beach, CA, December 2017.\n\n[8] Rong-En Fan, Kai-Wei Chang, Cho-Jui Hsieh, Xiang-Rui Wang, and Chih-Jen Lin. Liblinear: A library\n\nfor large linear classi\ufb01cation. Journal of machine learning research, 9(Aug):1871\u20131874, 2008.\n\n[9] George Marsaglia. Xorshift rngs. Journal of Statistical Software, Articles, 8(14), 2003.\n\n[10] Xiangrui Meng, Joseph Bradley, Burak Yavuz, Evan Sparks, Shivaram Venkataraman, Davies Liu, Jeremy\nFreeman, DB Tsai, Manish Amde, Sean Owen, Doris Xin, Reynold Xin, Michael J. Franklin, Reza Zadeh,\nMatei Zaharia, and Ameet Talwalkar. Mllib: Machine learning in apache spark. J. Mach. Learn. Res.,\n17(1):1235\u20131241, January 2016.\n\n[11] Gonzalo Gasca Meza. Samples for google cloud machine learning engine, 2017. https://github.com/\n\nGoogleCloudPlatform/cloudml-samples.\n\n[12] NVIDIA. Thrust, 2018. Software available at https://developer.nvidia.com/thrust.\n\n[13] Stack Over\ufb02ow. Use large dataset in tensor\ufb02ow, 2016. https://stackoverflow.com/questions/\n\n38087342/use-large-dataset-in-tensorflow.\n\n[14] Thomas Parnell, Celestine D\u00fcnner, Kubilay Atasu, Manolis Sifalakis, and Haralampous Pozidis. Tera-scale\n\ncoordinate descent on gpus. Future Generation Computer Systems, 0(0):0, 2018.\n\n[15] Thomas Parnell, Celestine D\u00fcnner, Kubilay Atasu, Manolis Sifalakis, and Haris Pozidis. Large-scale\nstochastic learning using gpus. In Proceedings of the IEEE International Parallel and Distributed Process-\ning Symposium Workshops, IPDPS\u201917, pages 419\u2013428, Orlando, FL, May 2017.\n\n[16] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer,\nR. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay.\nScikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825\u20132830, 2011.\n\n[17] NVIDIA Research. Cub, 2018. https://github.com/NVlabs/cub.\n\n[18] Virginia Smith, Simone Forte, Chenxin Ma, Martin Tak\u00e1\u02c7c, Michael I Jordan, and Martin Jaggi. CoCoA: A\n\nGeneral Framework for Communication-Ef\ufb01cient Distributed Optimization. JMLR, 18:1\u201349, 2018.\n\n[19] Rambler Digital\n\ncriteo-1tb-benchmark,\nrambler-digital-solutions/criteo-1tb-benchmark.\n\nSolutions.\n\n2017.\n\nhttps://github.com/\n\n10\n\n\f[20] Andreas\n\nSterbenz.\n\nscale,\n\nat\nusing-google-cloud-machine-learning-to-predict-clicks-at-scale.\n\n2017.\n\nUsing\n\ngoogle\n\ncloud machine\n\nclicks\nhttps://cloud.google.com/blog/big-data/2017/02/\n\nlearning\n\npredict\n\nto\n\n[21] Hsiang-Fu Yu, Cho-Jui Hsieh, Kai-Wei Chang, and Chih-Jen Lin. Large linear classi\ufb01cation when data\n\ncannot \ufb01t in memory. ACM Transactions on Knowledge Discovery from Data (TKDD), 5(4):23, 2012.\n\n[22] Hsiang-Fu Yu, Fang-Lan Huang, and Chih-Jen Lin. Dual coordinate descent methods for logistic regression\n\nand maximum entropy models. Machine Learning, 85(1):41\u201375, Oct 2011.\n\n[23] Huan Zhang and Cho-Jui Hsieh. Fixing the convergence problems in parallel asynchronous dual coordinate\ndescent. In Data Mining (ICDM), 2016 IEEE 16th International Conference on, pages 619\u2013628. IEEE,\n2016.\n\n11\n\n\f", "award": [], "sourceid": 185, "authors": [{"given_name": "Celestine", "family_name": "D\u00fcnner", "institution": "IBM Research"}, {"given_name": "Thomas", "family_name": "Parnell", "institution": "IBM Research"}, {"given_name": "Dimitrios", "family_name": "Sarigiannis", "institution": "IBM Research"}, {"given_name": "Nikolas", "family_name": "Ioannou", "institution": "IBM Research"}, {"given_name": "Andreea", "family_name": "Anghel", "institution": "IBM Research"}, {"given_name": "Gummadi", "family_name": "Ravi", "institution": "IBM Systems"}, {"given_name": "Madhusudanan", "family_name": "Kandasamy", "institution": "IBM Systems"}, {"given_name": "Haralampos", "family_name": "Pozidis", "institution": "IBM Research"}]}