{"title": "More Is Less: Learning Efficient Video Representations by Big-Little Network and Depthwise Temporal Aggregation", "book": "Advances in Neural Information Processing Systems", "page_first": 2264, "page_last": 2273, "abstract": "Current state-of-the-art models for video action recognition are mostly based on expensive 3D ConvNets. This results in a need for large GPU clusters to train and evaluate such architectures. To address this problem, we present an lightweight and memory-friendly architecture for action recognition that performs on par with or better than current architectures by using only a fraction of resources.\nThe proposed architecture is based on a combination of a deep subnet operating on low-resolution frames with a compact subnet operating on high-resolution frames, allowing for high efficiency and accuracy at the same time. We demonstrate that our approach achieves a reduction by 3~4 times in FLOPs and ~2 times in memory usage compared to the baseline. This enables training deeper models with more input frames under the same computational budget.  To further obviate the need for large-scale 3D convolutions, a temporal aggregation module is proposed to model temporal dependencies in a video at very small additional computational costs. Our models achieve strong performance on several action recognition benchmarks including Kinetics, Something-Something and Moments-in-time. The code and models are available at \\url{https://github.com/IBM/bLVNet-TAM}.", "full_text": "More Is Less: Learning Ef\ufb01cient Video\n\nRepresentations by Big-Little Network and Depthwise\n\nTemporal Aggregation\n\nQuanfu Fan\u2020,1, Chun-Fu (Richard) Chen\u2020,2, Hilde Kuehne1, Marco Pistoia2, David Cox1\n\n\u2020: Equal contribution\n\n1MIT-IBM Waston AI Lab, Cambridge, MA 02142, USA\n\n2IBM T.J. Waston Research Center, Yorktown Heights, NY 10598, USA\n\n{qfan, chenrich, pistoia}@us.ibm.com, {kuehne, david.d.cox}@ibm.com\n\nAbstract\n\nCurrent state-of-the-art models for video action recognition are mostly based on\nexpensive 3D ConvNets. This results in a need for large GPU clusters to train and\nevaluate such architectures. To address this problem, we present an lightweight\nand memory-friendly architecture for action recognition that performs on par with\nor better than current architectures by using only a fraction of resources. The\nproposed architecture is based on a combination of a deep subnet operating on\nlow-resolution frames with a compact subnet operating on high-resolution frames,\nallowing for high ef\ufb01ciency and accuracy at the same time. We demonstrate that our\napproach achieves a reduction by 3 \u223c 4 times in FLOPs and \u223c 2 times in memory\nusage compared to the baseline. This enables training deeper models with more\ninput frames under the same computational budget. To further obviate the need for\nlarge-scale 3D convolutions, a temporal aggregation module is proposed to model\ntemporal dependencies in a video at very small additional computational costs.\nOur models achieve strong performance on several action recognition benchmarks\nincluding Kinetics, Something-Something and Moments-in-time. The code and\nmodels are available at https://github.com/IBM/bLVNet-TAM.\n\n1\n\nIntroduction\n\nCurrent state-of-the-art approaches for video action recognition are based on convolutional neural\nnetworks (CNNs). These include the best performing 3D models, such as I3D [1] and ResNet3D [2],\nand some effective 2D models, such as Temporal Relation Networks (TRN) [3] and Temporal Shift\nModules (TSM) [4]. A CNN-based model usually considers a sequence of frames as input, obtained\nthrough either uniform or dense sampling from a video [1, 5]. In general, Longer input sequences\nyield better recognition results. However, one problem arising for a model requesting more input\nframes is that the GPU resources required for training and inference also signi\ufb01cantly increase in\nboth memory and time. For example, the top-performing I3D models [1] on the Kinetics [6] dataset\nwere trained with 64 frames on a cluster of 32 GPUs, and the non-local network [7] even uses 128\nframes as input. Another problem for action recognition is the lack of effective methods for temporal\nmodeling when moving away from 3D spatiotemporal convolutions. While 2D convolutional models\nare more resource-friendly than their 3D counterparts, they lack expressiveness over time and thus\ncannot take much bene\ufb01t from richer input data.\nIn this paper, we present an ef\ufb01cient and memory-friendly spatio-temporal representation for action\nrecognition, which enables training of deeper models while allowing for more input frames. The \ufb01rst\npart of our approach is inspired by the Big-Little-Net architecture (bLNet [8]). We propose a new\n\n33rd Conference on Neural Information Processing Systems (NeurIPS 2019), Vancouver, Canada.\n\n\fvideo architecture that has two network branches with different complexities: one branch processing\nlow-resolution frames in a very deep subnet, and another branch processing high-resolution frames\nin a compact subnet. The two branches complement each other through merging at the end of each\nnetwork layer. With such a design, our approach can process twice as many frames as the baseline\nmodel without compromising ef\ufb01ciency. We refer to this architecture as \u201cBig-Little-Video-Net\u201d\n(bLVNet).\nIn light of the limited ability of capturing temporal dependencies in bLVNet, we further develop an\neffective method to exploit temporal relations across frames by a so called \u201cDepthwise Temporal Ag-\ngregation Module\u201d (TAM). The method enables the exchange of temporal information between frames\nby weighted channel-wise aggregation. This aggregation is made learnable with 1\u00d71 depthwise\nconvolution, and implemented as an independent network module. The temporal aggregation module\ncan be easily integrated into the proposed network architecture to progressively learn spatio-temporal\npatterns in a hierarchical way. Moreover, the module is extremely compact and adds only negligible\ncomputational costs and parameters to bLVNet.\nOur main contributions lie in the following two interconnected aspects: (1) We propose a lightweight\nvideo architecture based on dual-path network to learn video features, and (2) we develop a temporal\naggregation module to enable effective temporal modeling without the need for computationally\nexpensive 3D convolutions.\nWe evaluate our approach on the Kinetics-400 [6], Something-Something [9] and Moments-in-\ntime [10] datasets. The evaluation shows that bLVNet-TAM successfully allows us to train action-\nclassi\ufb01cation models with deeper backbones (i.e., ResNet-101) as well as more (up to 64) input\nframes, using a single compute node with 8 Tesla V100 GPUs. Our comprehensive experiments\ndemonstrate that our approach achieves highly competitive results on all datasets while maintaining\nef\ufb01ciency. Especially, it establishes a new state-of-the-art result on Something-Something and\nMoments-in-time by outperforming previous approaches in the literature by a large margin.\n\n2 Related Work\n\nActivity classi\ufb01cation has always been a challenging research topic, with \ufb01rst attempts reaching back\nby almost two decades [11]; deep-learning architectures nowadays achieve tremendous recognition\nrates on various challenging tasks, such as Kinetics [1], ActivityNet [12], or Thumos [13].\nMost successful architectures in the \ufb01eld are usually based on the so-called two-stream model [14],\nprocessing a single RGB frame and optical-\ufb02ow input in two separate CNNs with a late fusion in the\nupper layers. Over the last years, many approaches extend this idea by processing a stack of input\nframes in both streams, thus extending the temporal window of the architecture form 1 to up to 128\ninput frames per stream. To further capture the temporal correlation in the input over time, those\narchitectures usually make use of 3D convolutions as, e.g., in I3D [1], S3D [15], and ResNet3D [2],\nusually leading to a large-scale parameter space to train.\nAnother way to capture temporal relations has been proposed by [5], [3], and [4]. Those architectures\nmainly build on the idea of processing videos in the form of multiple segments, and then fusing them\nat the higher layers of the networks. The \ufb01rst approach with this pattern was the so-called Temporal\nSegment Networks (TSN) proposed by Wang et al. [5]. The idea of TSN has been extended by\nTemporal Relation Networks (TRN) [3], which apply the idea of relational networks to the modeling\nof temporal relations between observations in videos. Another approach for capturing temporal\ncontexts has been proposed by Temporal Shift Modules (TSM) [4]. This approach shifts part of the\nchannels along the temporal dimension, thereby allowing for information to be exchanged among\nneighboring frames. More complex approaches have been tried as well, e.g. in the context of non-\nlocal neural networks [7]. Our temporal aggregation module is based on depthwise 1\u00d71 convolutions\nto capture temporal dependencies across frames effectively.\nSeparate convolutions are considered in approaches such as [15, 16] to reduce costly computation in\n3D convolutional models. More recently, SlowFast Network [17] uses a dual-pathway network to\nprocess a video at both slow and fast frame rates. The fast pathway is made lightweight, similar to\nLittle Net in our proposed architecture. However, our approach reduces computation based on both a\nlightweight architecture and low image resolution. Furthermore, the recent work Timeception [18]\napplies the concept of \u201cInception\" to temporal domain for capturing long-range temporal dependencies\n\n2\n\n\fin a video. The Timeception layers involve group convolutions at different time scales while our\nTAM layers only use depthwise convolution. As a result, the Timeception has signi\ufb01cantly more\nparameters than the TAM (10% vs. 0.1% of the total model parameters).\n\n3 Our Approach\n\nWe aim at developing ef\ufb01cient and effective video representations for video understanding. To address\nthe computational challenge imposed by the desired long input to a model, we propose a new video\narchitecture based on the Big-Little network (bLNet) [8] for learning video features.We \ufb01rst give a\nbrief recap of bLNet in Section 3.1. We then show, in Section 3.2, how to extend bLNet to an ef\ufb01cient\nvideo architecture that allows for seeing more frames with less computation and memory. An example\nof the proposed network architecture can be found in the supplementary material (Section A).\nTo make temporal modeling more effective in our approach, we further develop a temporal aggregation\nmodule (TAM) to capture short-term as well as long-term temporal dependencies across frames. Our\nmethod is implemented as a separate network module and integrated with the proposed architecture\nseamlessly to learn a hierarchical temporal representation for action recognition. We detail this\nmethod in Section 3.3.\n\n3.1 Recap of Big-Little Network\n\nThe Big-Little Net, abbreviated as bLNet in [8], is a CNN architecture for learning strong feature\nrepresentations by combining multi-scale image information. The bLNet processes an image at\ndifferent resolutions using a dual-path network, but with low computational loads based on a clever\ndesign. The key idea is to have a high-complexity subnet (Big-Net) along with a low-cost one\n(Little-Net) operate on the low-scale and high-scale parts of an image in parallel. By such a design,\nthe two subnets learn features complementary to each other while using less computation. The two\nbranches are merged at the end of each network layer to fuse the low-scale and high-scale information\nso as to form a stronger image representation. The bLNet approach demonstrates improvement of\nmodel ef\ufb01ciency and performance on both object and speech recognition, using popular architectures\nsuch as ResNet, ResNeXt and SEResNeXt. More details on bLNet can be found in the original\npaper. In this work, we mainly adopt bLResNet-50 and bLResNet-101 as backbone for our proposed\narchitecture.\n\n3.2 Big-Little Video Network as Video Representation\n\nWe describe our architecture in the context of 2D convolutions. However our approach is not speci\ufb01c\nto 2D convolutions and potentially extendable to any architecture based on 3D convolutions.\n\nFigure 1: Different architectures for action recognition. a) TSN [5] uses a shared CNN to process\neach frame independently, so there is no temporal interaction between frames. b) TSN-bLNet is a\nvariant of TSN that uses bLNet [8] as backbone. It is ef\ufb01cient, but still lacks temporal modeling. c)\nbLVNet feeds odd and even frames separately into different branches in bLNet. The branch merging\nat each layer (local fusion) captures short-term temporal dependencies between adjacent frames. d)\nbLVNet-TAM includes the proposed aggregation module, represented as a red box, which further\nempowers bLVNet to model long-term temporal dependencies across frames (global fusion).\n\n3\n\n!!!\"#$%&$%\u2019%()(*(+()(*!!(+(,!!(-(.!!!!!\"#$%&$%\u2019%!()!!!(*!!!(+!!\"#$%&$%\u2019%()(*!!(-(.!!(+(,!!!!!\"#$%&$%\u2019%\"#$%&\u2019(#$%&\u2019)(*\u2019+,-#$(*.\u2019+,/#$(*.\u2019+,)%01!!/&0\u20191234$#5&6#5&42(7&341#8214(\u2019%9#$6#5&42(7&3401#:214(\u2019%9#$;90<6&7=9771&<6&7;32$8>4?&309$0!3&59879#$(9@32?&\fThe approach of Temporal Segment Networks (TSN) [5] provides a generic framework for learning\nvideo representations. With a shared 2D ConvNet as backbone, TSN performs frame-level predictions\nand then aggregates the results into a \ufb01nal video-level prediction (Fig. 1a)). The framework of TSN is\nef\ufb01cient and has been successfully adopted by some recent approaches for action recognition such as\nTRN [3] and TSM [4]. Given its ef\ufb01ciency, we also choose TSN as the underlying video framework\nfor our work.\nLet F = {ft|t = 1\u00b7\u00b7\u00b7 n} be a set of sampled input frames from a video. We divide F into two\ngroups, namely odd frames Fodd = {fk \u2208 F| mod (k, 2) (cid:54)= 0} at half of the input image resolution,\nand even frames Feven = {fk \u2208 F| mod (k, 2) = 0} at the input image resolution. For convenience,\nfrom now on, Fodd is referred to as big frames and Feven as little frames. Note that big branch can\ntake either of a pair of frames as input and the other frame goes to the little branch.\nIn TSN, all input frames are ordered as a batch of size n, where the tth element corresponds to the\ntth frame. We denote the input and output feature maps of the tth frame at the kth layer of the model\nby xk\nThe bLNet can be directly plugged into TSN as the backbone network for learning video-level\nrepresentation. We refer to this architecture as TSN-bLNet to differentiate it from the vanilla TSN\n(Fig. 1b)). This network fully enjoys the ef\ufb01ciency of bLNet, cutting the computational costs down\nby 1.6 \u223c 2 times according to [8]. Mathematically, the output yt can be written as\n\nt \u2208 RC\u00d7W\u00d7H, respectively. Whenever possible, we omit k for clarity.\n\nt \u2208 RC\u00d7W\u00d7H and yk\n\nyt = F(netB([xt]1/2) + netL(xt), \u03b8t).\n\n(1)\nHere [\u00b7]s is an operator scaling a tensor up or down by a factor of s in the spatial domain; netB and\nnetL are the Big-Net and Little-Net in the bLNet aforementioned; and \u03b8t are the model parameters.\nFollowing [8], F indicates an additional residual block applied after merging the big and little\nbranches to stabilize and enhance the combined feature representation.\nThe architecture described above only learns features from a single frame, so there are no interactions\nbetween frames. Alternatively, we can feed the odd and even frames separately into the big and little\nbranches so that each branch obtains complementary information from different frames. This idea is\nillustrated in Fig. 1c) and the output yt in this case can be expressed by\n\n(cid:26)F(netB((cid:98)xt(cid:99)1/2) + netL(xt+1), \u03b8t),\n\nyt =\n\nyt\u22121,\n\nif mod (t, 2) (cid:54)= 0\notherwise\n\n(2)\n\nWhile the modi\ufb01cation proposed above is simple, it leads to a new video architecture, which is\ncalled Big-Little-Video-Net, or bLVNet for short. The bLVNet makes two distinct differences from\nTSN-bLNet. Firstly, without increasing any computation, it can take input frames two times as many\nas TSN-bLNet. We shall demonstrate the bene\ufb01t of leveraging more frames for temporal modeling in\nSection 4. Furthermore, the bLVNet has 1.5 \u223c 2.0\u00d7 fewer FLOPs than TSN while seeing frames\ntwice as many as TSN, thanks to the ef\ufb01ciency of the dual-path network. Secondly, the merging of\nthe two branches in bLVNet now happens on two different frames carrying temporal information.\nWe call this type of temporal interaction by local fusion, since it only captures temporal relations\nbetween two adjacent frames. In spite of that, local fusion gives rise to a signi\ufb01cant performance\nboost for recognition, as shown later in Section 4.3.\n\n3.3 Temporal Aggregation Module\n\nTemporal modeling is a challenging problem for video understanding. Theoretically, adding a\nrecurrent layer such as LSTM [19] on top of a 2D ConvNet seems like a promising means to capture\ntemporal ordering and long-term dependencies in actions. Nonetheless, such approaches are not\npractically competent with 3D ConvNets [1], which use spatio-temporal \ufb01lters to learn hierarchical\nfeature representations. One issue with 3D models is that they are heavy in parameters and costly\nin computation, making them hard to train. Even though some approaches like S3D [15] and\nR(2+1)D [16] alleviates this issue by separating a 3D convolution \ufb01lter into a 2D spatial component\nfollowed by a 1D temporal component, they are in general still more expensive than 2D ConvNet\nmodels.\nWith the ef\ufb01cient bLVNet architecture described above, our goal is to further improve its spatio-\ntemporal representation by effective temporal modeling. The local fusion in bLVNet only exploits\ntemporal relations between neighbored frames. To address this limitation, we develop a method\n\n4\n\n\fto capture short-term as well as long-term dependencies across frames. Our basic idea is to fuse\ntemporal information at each time instance by weighted channel-wise aggregation. As detailed below,\nthis idea can be ef\ufb01ciently implemented as a network module to progressively learn spatio-temporal\npatterns in a hierarchical way.\nLet yt be the output (i.e. neural activation) of the tth frame ft at a layer of the network (see Eq. 2).\nTo model the temporal dependencies between ft and its neighbors, we aggregate the activations of\nall the frames within a temporal range r around ft. A weight is learned for each channel of the\nactivations to indicate its relevance. Speci\ufb01cally, the aggregation results can be written as\n\nj=(cid:98)r/2(cid:99)(cid:88)\n\nj=\u2212(cid:98)r/2(cid:99)\n\n\u02c6yt = ReLU (\n\nwj \u2297 yt+j),\n\n(3)\n\nwhere \u2297 indicates the channel-wise multiplication and wj \u2208 RC is the weights. The \u2297 is de\ufb01ned\nas: for a vector v = [v1 v2 \u00b7\u00b7\u00b7 vC] and a tensor M = [m1 m2 \u00b7\u00b7\u00b7 mC] with C feature channels,\nv \u2297 M = [v1 \u2217 m1 v2 \u2217 m2 \u00b7\u00b7\u00b7 vC \u2217 mC].\nWe implement the temporal aggregation as a network module (Fig. 2). It involves three steps as\nfollows,\n\nr \u00d7 n;\n\n1. apply 1\u00d71 depthwise convolution r times to n input tensors to form an output matrix of size\n2. shift the ith row left (or right) by |i \u2212 (cid:98)r/2(cid:99)| positions if i > (cid:98)r/2(cid:99) (or i \u2264 (cid:98)r/2(cid:99)) and if\n\nneeded, pad leading or trailing zero tensors in the front or at the end;\n\n3. perform temporal aggregation along the column to generate the output.\n\nThe aggregation module(TAM), highlighted as a red box in Fig. 1d), is inserted as a separate layer after\nthe local temporal fusion in the bLVNet, resulting in the \ufb01nal bLVNet-TAM architecture. Obviously\nnone of the steps in the implementation above involve costly computation, so the module is fairly\nfast. A node in the network initially only sees r \u2212 1 neighbors. As the network goes deeper, the\namount of context that the node involves in the input grows quickly, similar to how the receptive\n\ufb01eld of a neuron is enlarged in a CNN. In such a manner, long-range temporal dependencies are thus\npotentially captured. For this reason, the temporal aggregation is also called global temporal fusion\nhere, as opposed to the local temporal fusion discussed above.\nThe work of TSM [4] has also applied temporal shifting to swap feature channels between neighboring\nframes. In such a case, TSM can be treated as a special case of our method where the weights are\nempirically set rather than learned from data. In Section 4.3, we demonstrate that the proposed TAM\nis more effective than TSM for temporal modeling under different video architectures. TAM is also\nrelated to S3D [15] and R(2+1)D [16] in that TAM is independent of spatial convolutions. However,\nTAM is based on depthwise convolution, thus has fewer parameters and less computation than S3D\nand R(2+1)D.\nThe TAM can also be integrated into 3D convolutions such as C3D [20] and I3D [1] to further enhance\nthe temporal modeling capability that already exists in these models. Due to the difference in how\ntemporal data is presented between 2D-based and 3D-based models, the temporal shifting now needs\nto operate on feature channels within a tensor instead of on tensors themselves.\n\nFigure 2: Temporal aggregation module (TAM). The TAM takes as input a batch of tensors, each of\nwhich is the activation of a frame, and produces a batch of tensors with the same order and dimension.\nThe module consists of three operations: 1) 1\u00d71 depthwise convolutions to learn a weight for each\nfeature channel; 2) temporal shifts (left or right direction indicated by the smaller arrows; the white\ncubes are padded zero tensors.); and 3) aggregation by summing up the weighted activations from 1).\n\n5\n\n!\"#!\"#$%&\u2019&()*#%+,-.*/0\"123*4#0567(8+-9%-\":3*4#0567(;::5*:6%-0\"\f4 Experiments\n\n4.1 Experimental Setup\n\nDatasets. We evaluate our approach on three large-scale datasets for video recognition, including\nthe widely used Something-Something (Version 1 and Version 2) [9], Kinetics-400 [6] and the\nrecent Moments-in-time dataset [10]. They are herein referred to as SS-V1, SS-V2, Kinetics-400 and\nMoments, respectively.\nSomething-Something is a dataset containing videos of 174 types of prede\ufb01ned human-object\ninteractions with everyday objects. The version 1 and 2 include 108k and 220k videos, respectively.\nThis dataset focuses on human-object interactions in a rather simple setup with no scene contexts to be\nexploited for recognition. Instead temporal relationships are as important as appearance for reasoning\nabout the interactions. Because of this, the dataset serves as a good benchmark for evaluating the\nef\ufb01cacy of temporal modeling, such as proposed in our approach.\nKinetics-400 [6] has emerged as a standard benchmark for action recognition after UCF101 [21] and\nHMDB [22], but on a signi\ufb01cantly larger scale. The dataset consists of 240k training videos and 20k\nvalidation videos, with each video trimmed to around 10 seconds. It has a total of 400 human action\ncategories.\nMoments-in-time [10] is a recent collection of one million labeled videos, involving actions from\npeople, animals, objects or natural phenomena. It has 339 classes and each video clip is trimmed to 3\nseconds long.\nData Augmentation. During training, we follow the data augmentation used in TSN [5] to augment\nthe video with different sizes spatially and \ufb02ip the video horizontally with 50% probability. Further-\nmore, since our models are \ufb01netuned on pretrained ImageNet, we normalize the data with the mean\nand standard deviation of the ImageNet images. The model input is formed by uniform sampling,\nwhich \ufb01rst divides a video into n uniform segments and then selects one random frame from each\nsegment as the input.\nDuring inference, we resize the smaller side of an image to 256 and then crop a centered 224\u00d7224\nregion. The center frame of each segment in uniform sampling is picked as the input. On Something-\nSomething and Moments, our results are based on the single-crop and single-clip setting. On\nKinetics-400, we use the common practice of multi-crop and multi-clip for evaluation.\nTraining Details. Since all the three datasets are large-scale, we train the models in a progressive\nway. For each type of backbone (for example, bLResNet-50), we \ufb01rst \ufb01netune a base model on\nImageNet with a minimum input length (i.e. 8\u00d72 in our case) using 50 epochs. We adopt the Nesterov\nmomentum optimizer with an initial weight of 0.01, a weight decay of 0.0005 and a momentum of\n0.9. We then \ufb01netune a new model with longer input (for example, 16\u00d72) on top of the corresponding\nbase model, but with 25 epochs only. In this case, the initial learning rate is set to 0.01 on Something-\nSomething and 0.005 on Kinetics and Moments. The learning rate is decreased by a factor of 10 at\nthe 10-th and 20-th epoch, respectively.\nThis strategy allows to signi\ufb01cantly reduce the training time needed for all the models evaluated in\nour experiments. All our models were trained on a server with 8 GPU cards and a total of 128G GPU\nmemory. We set the total batch size to 64 whenever possible. For models that require more memory\nto train, we adjust the batch size accordingly to the maximum number allowed.\n\n4.2 Main Results\n\nSomething-Something. We \ufb01rst report our results on the validation set of the Something-Something\ndatasets in Table 1 and Table 2. With a moderately deep backbone bLResNet-50, our approach outper-\nforms all 3D models on SS-V1 while using much fewer input frames (8\u00d72) and being substantially\nmore ef\ufb01cient. TSM [4] was the previously best approach on Something-Something. Under the same\nbackbone (i.e. ResNet-50), our approach is better than TSM on both SS-V1 and SS-V2 while being\nmore ef\ufb01cient (i.e our 8x2 model has 1.4 times fewer FLOPs than a 8-frame TSM model).\nWhen empowered with a stronger backbone bLResNet-101, our approach achieves even better results\nat 32\u00d72 frames (53.1% top-1 accuracy on SS-V1, and 65.2% on SS-V2), establishing a new state-\nof-the-art on Something-Something. Notably, these results while based on RGB information only,\n\n6\n\n\fTable 1: Recognition Accuracy of Various Models on Something-Something-V1 (SS-V1).\n\nModel\n\nBackbone\n\nPretrain\n\nFrames Modality\n\nParam (106)\n\nFLOPs (109)\n\nVal\n\nTest\n\nTop-1 (%) Top-5 (%) Top-1 (%)\n\nI3D [1]\n\nNL I3D + GCN [23]\n\nS3D [15]\n\nECO-LiteEn [24]\n\nTSN [5]\nTRN [3]\n\nTSM [4]\n\nbLVNet-TAM\n\nInception\nResNet-50\nInception\n\nBNInception+ResNet18\n\nBNInception\nBNInception\nBNInception\nResNet-50\nResNet-50\nResNet-50\nbLResNet-50\nbLResNet-50\nbLResNet-101\nbLResNet-101\nbLResNet-101\nbLResNet-101\n\nImageNet\nImageNet\nImageNet\nImageNet\nImageNet\nImageNet\nImageNet\nKinetics\nKinetics\nKinetics\nImageNet\n\nSS-V1\n\nImageNet\n\nSS-V1\nSS-V1\nSS-V1\n\n64\n\n32+32\n\n64\n92\n8\n8\n8+8\n8\n16\n\n16+16\n8\u00d72\n16\u00d72\n8\u00d72\n16\u00d72\n24\u00d72\n32\u00d72\n\nRGB\nRGB\nRGB\nRGB\nRGB\nRGB\n\nRGB+Flow\n\nRGB\nRGB\n\nRGB+Flow\n\nRGB\nRGB\nRGB\nRGB\nRGB\nRGB\n\n12.7\n303\n8.77\n150\n10.7\n18.3\n\u2212\n24.3\n24.3\n\u2212\n25.0\n25.0\n40.2\n40.2\n40.2\n40.2\n\n111\n62.2\n66\n267\n16\n16\n\u2212\n33\n65\n\u2212\n23.8\n47.7\n32.1\n64.3\n96.4\n128.6\n\n45.8\n46.1\n47.3\n46.4\n19.5\n34.4\n42.0\n45.6\n47.2\n52.6\n46.4\n48.4\n47.8\n49.6\n52.2\n53.1\n\n76.5\n76.8\n78.1\n\u2212\n\u2212\n\u2212\n\u2212\n74.2\n77.1\n81.9\n76.6\n78.8\n78.0\n79.8\n81.8\n82.9\n\n27.2\n\u2212\n\u2212\n42.3\n\u2212\n33.6\n40.7\n\n46.0\n50.7\n\u2212\n\u2212\n\u2212\n\u2212\n\u2212\n48.9\n\nTable 2: Recognition Accuracy of Various Models on Something-Something-V2 (SS-V2).\n\nModel\n\nBackbone\n\nPretrain\n\nFrames Modality\n\nParam (106)\n\nFLOPs (109)\n\nVal\n\nTest\n\nTop-1 (%) Top-5 (%) Top-1 (%) Top-5 (%)\n\nTRN [3]\n\nTSM [4]\n\nbLVNet-TAM\n\nBNInception\nBNInception\nResNet-50\u2020\nResNet-50\u2020\nResNet-50\nbLResNet-50\nbLResNet-50\nbLResNet-101\nbLResNet-101\nbLResNet-101\nbLResNet-101\nbLResNet-101\u2217\n\nImageNet\nImageNet\nKinetics\nKinetics\nKinetics\nImageNet\n\nSS-V2\n\nImageNet\n\nSS-V2\nSS-V2\nSS-V2\nSS-V2\n\n8\n8\n8\n16\n\u2212\n8\u00d72\n16\u00d72\n8\u00d72\n16\u00d72\n24\u00d72\n32\u00d72\n32\u00d72\n\nRGB\n\nRGB+Flow\n\nRGB\nRGB\n\nRGB+Flow\n\nRGB\nRGB\nRGB\nRGB\nRGB\nRGB\n\nRGB+Flow\n\n18.3\n36.6\n24.3\n24.3\n\u2212\n25.0\n25.0\n40.2\n40.2\n40.2\n40.2\n\u2212\n\n16\n32\n33\n65\n\u2212\n23.8\n47.7\n32.1\n64.3\n96.4\n128.6\n\u2212\n\n48.8\n55.5\n58.9\n61.4\n66.0\n59.1\n61.7\n60.2\n61.9\n64.0\n65.2\n68.5\n\n77.6\n83.1\n85.5\n87.0\n90.5\n86.0\n88.1\n87.1\n88.4\n89.8\n90.3\n91.4\n\n50.9\n56.2\n\u2212\n\u2212\n66.6\n\u2212\n\u2212\n\u2212\n\u2212\n\u2212\n\u2212\n67.1\n\n79.3\n83.2\n\u2212\n\u2212\n91.3\n\u2212\n\u2212\n\u2212\n\u2212\n\u2212\n\u2212\n91.4\n\n\u2020 : using their pretrained models and code to evaluate under the 1-crop and 1-clip setting for fair comparison\n\u2217 : model ensemble of RGB and Flow model, each is evaluated with 3 crops and 10 clips and uses 256 as the shorter side.\n\nare superior to those obtained from the best two-stream models at no more computational cost. This\nstrongly demonstrates the effectiveness of our approach for temporal modeling. We further evaluated\nour models on the test set of Something-Something. Our results are consistently better than the best\nresults reported by the other approaches in comparison including 2-stream models.\nKinetics-400. Kinetics-400 is one of the most popular benchmarks for action recognition. Currently\nthe best-performed models on this dataset are all based on 3D Convolutions. However, it has been\nshown in the literature that temporal ordering in this dataset does not seem to be as crucial as RGB\ninformation for recognition. For example, as experimented in S3D [15], the model trained on normal\ntime-order data performs well on the time-reversed data on Kinetics. In accordance to this, our\napproach (3 crops and 3 clips) mainly performs on par with or better than the current large-scale\narchitectures, but without outperforming them as clearly as on the Something-Something datasets,\nwhere the temporal relations are more essential for an overall understanding of the video content.\n\nTable 3: Recognition Accuracy of Various Models on Kinetics-400 (RGB-only).\n\nNet\n\nBackbone\n\nFLOPs (109) Top-1 (%) Top-5 (%)\n\nSTC [25]\n\nARTNet [26]\n\nC3D [26]\nI3D [1]\nS3D [15]\n\nR(2+1)D [16]\n\nSlowFast-4\u00d716 [17]\n\nTSN [5]\n\nTSM-8 [4]\nTSM-16 [4]\n\nbLVNet-TAM-8\u00d72\nbLVNet-TAM-16\u00d72\nbLVNet-TAM-24\u00d72\n\nResNeXt-101\n\nResNet-18\nResNet-18\nInception\nInception\nResNet-34\nResNet-50\nInceptionV3\n\nResNet-50\nResNet-50\nbLResNet-50\nbLResNet-50\nbLResNet-50\n\nECO-LiteEn [24]\n\nBNInception+ResNet18\n\n\u2212\n\n\u2212\n\u2212\n\n23.5\u00d7250\n19.6\u00d7250\n108\u00d7N/A\n\n267\n\n36.1\u00d730\n142.8\u00d710\n42.7\u00d730\n85.4\u00d730\n31.1\u00d79\n62.3\u00d79\n93.4\u00d79\n\n68.7\n69.2\n65.6\n71.1\n72.2\n72.0\n75.6\n72.5\n70.7\n74.1\n74.7\n71.0\n72.0\n73.5\n\n88.5\n88.3\n85.7\n89.3\n90.6\n90.0\n92.1\n\u2212\n-\n91.2\n\u2212\n89.8\n90.6\n91.2\n\nPretrain\nNone\nNone\nNone\n\nImageNet\nImageNet\n\nNone\nNone\n\nImageNet\nImageNet\nImageNet\nImageNet\nImageNet\nKinetics\nKinetics\n\n7\n\n\fTable 4: Recognition Accuracy of Various Models on Moments-in-time.\n\nNet\n\nBackbone\n\nPretrain\n\nFrames Modality\nAudio\nRGB\n\n\u2212\n16\n\nRGB+Flow\n\n16+16\n\n16\n16\n\u2212\n8\u00d72\n16\u00d72\n\nRGB\nRGB\n\u2212\nRGB\nRGB\n\nTop-1 (%) Top-5 (%)\n\n7.60\n24.1\n25.3\n28.3\n29.5\n31.2\n31.2\n31.4\n\n18.0\n49.1\n50.1\n53.9\n56.1\n57.7\n58.3\n59.3\n\nSoundNet [10]\n\nTSN [10]\nTSN [10]\nTRN [10]\nI3D [10]\n\nBNInception\nBNInception\n\nInception\nResNet-50\n\n\u2212\n\n\u2212\n\nImageNet\n\nImageNet\n\n\u2212\n\u2212\n\u2212\n\u2212\n\nEnsemble [10]\nbLVNet-TAM bLResNet-50\n\nImageNet\nbLResNet-50 Moments\n\nFigure 3: Number of input frames v.s. model accuracy and memory usage. (a) A longer input sequence\nyields better recognition in our proposed bLVNet-TAM on the Something-Something dataset [9], but\nnot in TSN [5] due to limited temporal modeling ability. (b) Compared to TSN, bLVNet-TAM reduces\nmemory usage by \u223c2 times under the same number of input frames.\n\nMoments. We \ufb01nally evaluate the proposed architecture on the Moments dataset [10], a large-scale\naction dataset with about three times more training samples than Kinetics-400. Since Moments\nis relatively new and results reported on it are limited, we only compare our results with those\nreported in the Moments paper [10]. As can been seen from Table 4, our approach outperforms all\nthe single-stream models as well as the ensemble one. We hope our models provide stronger baseline\nresults for future reference on this challenging dataset.\nIt is also noted that our model trained with 16 \u00d7 2 frames only produces slightly better top-1 accuracy\nthan the model trained with 8 \u00d7 2 frames. We speculate that this has to do with the fact that the\nMoments clips are only as short as 3 seconds and that there is only a limited impact in choosing a\n\ufb01ner temporal granularity on this dataset.\n\n4.3 Ablation Studies\n\nIn this section, we conduct ablation studies to provide more insights about our main ideas.\nIs temporal aggregation effective?. We validate the ef\ufb01cacy of the proposed temporal aggregation\nmodule (TAM), which is considered as a global fusion method (Section 3.3). Local fusion here is\nreferred to the branch merging in the dual path network (Section 3.2). We compare TAM with the\ntemporal shift module used in TSM [4] in Table 5 under two different video architectures: TSN and\nbLVNet proposed in this work. TAM demonstrates clear advantages over TSM, outperforming TSM\nby over 2% under both architectures. Interestingly, with the here proposed bLVNet baseline with\nlocal temporal fusion almost doubles the performance of a TSN baseline, improving the accuracy\nfrom 17.4% to 33.6%. On top of that, TAM boosts the performance by another 13% in both cases,\nsuggesting that TAM is complementary to local fusion. This further con\ufb01rms the signi\ufb01cance of\ntemporal reasoning on the Something-Something dataset.\nDoes seeing more frames help?. One of the main contribution of this work is an ef\ufb01cient video\narchitecture that makes it possible to train deeper models with more input frames using moderate\nGPU resources. Fig. 3a) shows consistent improvement of our approach on SS-V1 as the number of\ninput frames increases. A similar trend in our results can be observed on Kinetics-400 in Table 3. On\nthe other hand, the almost \ufb02attened line from TSN suggests that a model without effective temporal\nmodeling cannot take much of the bene\ufb01t from longer input frames.\n\n8\n\na)b)\fMemory Usage. We compare the memory usage between our approach based on bLResNet-50 and\nTSN based on ResNet-50. As shown in Fig. 3b), our approach is more memory friendly than TSN,\nachieving a saving of \u223c2 times at the same number of input frames. The larger batch size allowed for\ntraining under the same computational budget is critical for our approach to obtain better models and\nreduce training time.\n\n5 Conclusion\n\nWe presented an ef\ufb01cient and memory-friendly\nvideo architecture for learning video represen-\ntations. The proposed architecture allows for\ntwice as many input frames as the baseline while\nusing less computation and memory. This en-\nables training of deeper models with richer input\nunder the same GPU resources. We further de-\nveloped a temporal aggregation method to cap-\nture temporal dependencies effectively across\nframes. Our models achieve strong performance on several action recognition benchmarks, and\nestablish a state-of-the-art on the Something-Something dataset.\n\nBackbone\nResNet-50\nResNet-50\nResNet-50\nbLResNet-50\nbLResNet-50\nbLResNet-50\n\nTable 5: Temporal Modeling on SS-V1.\n\nLocal Fusion Global Fusion Top-1 (%)\n\nNet\n\nTSN\n\nbLVNet\n\nNone\nNone\nNone\n(cid:88)\n(cid:88)\n(cid:88)\n\nNone\nTSM\nTAM\nNone\nTSM\nTAM\n\n17.4\n43.4\n46.1\n33.6\n44.2\n46.4\n\nReferences\n[1] Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset.\nIn proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6299\u20136308,\n2017.\n\n[2] Kensho Hara, Hirokatsu Kataoka, and Yutaka Satoh. Learning spatio-temporal features with 3d residual\nnetworks for action recognition. In Proceedings of the IEEE International Conference on Computer Vision,\npages 3154\u20133160, 2017.\n\n[3] Bolei Zhou, Alex Andonian, Aude Oliva, and Antonio Torralba. Temporal relational reasoning in videos.\n\nIn Proceedings of the European Conference on Computer Vision (ECCV), pages 803\u2013818, 2018.\n\n[4] Ji Lin, Chuang Gan, and Song Han. Temporal shift module for ef\ufb01cient video understanding. arXiv\n\npreprint arXiv:1811.08383, 2018.\n\n[5] Limin Wang, Yuanjun Xiong, Zhe Wang, Yu Qiao, Dahua Lin, Xiaoou Tang, and Luc Van Gool. Temporal\nsegment networks: Towards good practices for deep action recognition. In Proceedings of the European\nConference on Computer Vision (ECCV). Springer, 2016.\n\n[6] Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan,\nFabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics human action video dataset. arXiv\npreprint arXiv:1705.06950, 2017.\n\n[7] Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In The\n\nIEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.\n\n[8] Chun-Fu (Richard) Chen, Quanfu Fan, Neil Mallinar, Tom Sercu, and Rogerio Feris. Big-little net: An\nef\ufb01cient multi-scale feature representation for visual and speech recognition. In International Conference\non Learning Representations, 2019.\n\n[9] Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne Westphal,\nHeuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitag, et al. The\" something\nsomething\" video database for learning and evaluating visual common sense. In ICCV, 2017.\n\n[10] Mathew Monfort, Alex Andonian, Bolei Zhou, Kandan Ramakrishnan, Sarah Adel Bargal, Yan Yan, Lisa\nBrown, Quanfu Fan, Dan Gutfreund, Carl Vondrick, et al. Moments in time dataset: one million videos for\nevent understanding. IEEE transactions on pattern analysis and machine intelligence, 2019.\n\n[11] J.K. Aggarwal and M.S. Ryoo. Human activity analysis: A review. ACM Comput. Surv., 43(3), 2011.\n\n[12] Bernard Ghanem Fabian Caba Heilbron, Victor Escorcia and Juan Carlos Niebles. Activitynet: A large-\nscale video benchmark for human activity understanding. In The IEEE Conference on Computer Vision\nand Pattern Recognition (CVPR), 2015.\n\n9\n\n\f[13] Y.-G. Jiang, J. Liu, A. Roshan Zamir, G. Toderici, I. Laptev, M. Shah, and R. Sukthankar. THUMOS\nchallenge: Action recognition with a large number of classes. http://crcv.ucf.edu/THUMOS14/,\n2014.\n\n[14] Karen Simonyan and Andrew Zisserman. Two-stream convolutional networks for action recognition in\n\nvideos. In Neural Information Processing System (NIPS), 2014.\n\n[15] Saining Xie, Chen Sun, Jonathan Huang, Zhuowen Tu, and Kevin Murphy. Rethinking spatiotemporal\nIn Proceedings of the European\n\nfeature learning: Speed-accuracy trade-offs in video classi\ufb01cation.\nConference on Computer Vision (ECCV), pages 305\u2013321, 2018.\n\n[16] Du Tran, Heng Wang, Lorenzo Torresani, Jamie Ray, Yann LeCun, and Manohar Paluri. A closer look at\nspatiotemporal convolutions for action recognition. In Proceedings of the IEEE conference on Computer\nVision and Pattern Recognition, pages 6450\u20136459, 2018.\n\n[17] Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video\n\nrecognition. arXiv preprint arXiv:1812.03982, 2018.\n\n[18] Noureldien Hussein, Efstratios Gavves, and Arnold W.M. Smeulders. Timeception for complex action\nrecognition. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.\n\n[19] Jeffrey Donahue, Lisa Anne Hendricks, Sergio Guadarrama, Marcus Rohrbach, Subhashini Venugopalan,\nKate Saenko, and Trevor Darrell. Long-term recurrent convolutional networks for visual recognition and\ndescription. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2015.\n\n[20] Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri. Learning Spatiotemporal\nFeatures With 3D Convolutional Networks. In The IEEE International Conference on Computer Vision\n(ICCV), December 2015.\n\n[21] Khurram Soomro, Amir Roshan Zamir, Mubarak Shah, Khurram Soomro, Amir Roshan Zamir, and\n\nMubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv, 2012.\n\n[22] H. Kuehne, H. Jhuang, E. Garrote, T. Poggio, and T. Serre. HMDB: a large video database for human\n\nmotion recognition. In Proceedings of the International Conference on Computer Vision (ICCV), 2011.\n\n[23] Xiaolong Wang and Abhinav Gupta. Videos as space-time region graphs. In The European Conference on\n\nComputer Vision (ECCV), September 2018.\n\n[24] Mohammadreza Zolfaghari, Kamaljeet Singh, and Thomas Brox. Eco: Ef\ufb01cient convolutional network for\nonline video understanding. In Proceedings of the European Conference on Computer Vision (ECCV),\npages 695\u2013712, 2018.\n\n[25] Ali Diba, Mohsen Fayyaz, Vivek Sharma, M. Mahdi Arzani, Rahman Yousefzadeh, Juergen Gall, and\nLuc Van Gool. Spatio-temporal channel correlation networks for action classi\ufb01cation. In The European\nConference on Computer Vision (ECCV), September 2018.\n\n[26] Limin Wang, Wei Li, Wen Li, and Luc Van Gool. Appearance-and-relation networks for video classi\ufb01cation.\n\nIn The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.\n\n10\n\n\f", "award": [], "sourceid": 1336, "authors": [{"given_name": "Quanfu", "family_name": "Fan", "institution": "MIT-IBM Watson AI Lab"}, {"given_name": "Chun-Fu (Richard)", "family_name": "Chen", "institution": "IBM Research"}, {"given_name": "Hilde", "family_name": "Kuehne", "institution": "University of Bonn"}, {"given_name": "Marco", "family_name": "Pistoia", "institution": "IBM Research"}, {"given_name": "David", "family_name": "Cox", "institution": "MIT-IBM Watson AI Lab"}]}