{"title": "Deep Alternative Neural Network: Exploring Contexts as Early as Possible for Action Recognition", "book": "Advances in Neural Information Processing Systems", "page_first": 811, "page_last": 819, "abstract": "Contexts are crucial for action recognition in video. Current methods often mine contexts after extracting hierarchical local features and focus on their high-order encodings. This paper instead explores contexts as early as possible and leverages their evolutions for action recognition. In particular, we introduce a novel architecture called deep alternative neural network (DANN) stacking alternative layers. Each alternative layer consists of a volumetric convolutional layer followed by a recurrent layer. The former acts as local feature learner while the latter is used to collect contexts. Compared with feed-forward neural networks, DANN learns contexts of local features from the very beginning. This setting helps to preserve hierarchical context evolutions which we show are essential to recognize similar actions. Besides, we present an adaptive method to determine the temporal size for network input based on optical flow energy, and develop a volumetric pyramid pooling layer to deal with input clips of arbitrary sizes. We demonstrate the advantages of DANN on two benchmarks HMDB51 and UCF101 and report competitive or superior results to the state-of-the-art.", "full_text": "Deep Alternative Neural Network: Exploring\n\nContexts as Early as Possible for Action Recognition\n\nJinzhuo Wang, Wenmin Wang, Xiongtao Chen, Ronggang Wang, Wen Gao\u2020\n\u2020School of Electronics Engineering and Computer Science, Peking University\n\nSchool of Electronics and Computer Engineering, Peking University\n\njzwang@pku.edu.cn, wangwm@ece.pku.edu.cn\n\ncxt@pku.edu.cn, rgwang@ece.pku.edu.cn, wgao@pku.edu.cn\n\nAbstract\n\nContexts are crucial for action recognition in video. Current methods often mine\ncontexts after extracting hierarchical local features and focus on their high-order\nencodings. This paper instead explores contexts as early as possible and lever-\nages their evolutions for action recognition. In particular, we introduce a novel\narchitecture called deep alternative neural network (DANN) stacking alternative\nlayers. Each alternative layer consists of a volumetric convolutional layer followed\nby a recurrent layer. The former acts as local feature learner while the latter is\nused to collect contexts. Compared with feed-forward neural networks, DANN\nlearns contexts of local features from the very beginning. This setting helps to\npreserve hierarchical context evolutions which we show are essential to recognize\nsimilar actions. Besides, we present an adaptive method to determine the temporal\nsize for network input based on optical \ufb02ow energy, and develop a volumetric\npyramid pooling layer to deal with input clips of arbitrary sizes. We demonstrate\nthe advantages of DANN on two benchmarks HMDB51 and UCF101 and report\ncompetitive or superior results to the state-of-the-art.\n\n1\n\nIntroduction\n\nContexts contribute semantic clues for action recognition in video. Current leading convolutional\nneural networks (CNNs) [13, 22, 31] and its shifted version 3D CNNs [11, 28, 29] often aggregate\ncontexts in the late stage. More precisely, in the \ufb01rst layer of a typical CNN, receptive \ufb01eld (RF)\nstarts at the kernel size which is usually small and the outputs only extract local features. As the layer\ngoes deeper, RF expands and contexts start to be involved. These models need to be very deep [32] to\npreserve rich context topologies and reach competitive trajectory-based works [16, 19, 20, 30]. We\nspeculate this is the main reason that going deeper with convolutions achieves better performance\non many visual recognition tasks [23, 26]. However, it is not wise to simply increase layer number\ndue to parameter burden. Besides, these models do not embed context evolutions of local features in\nthe forward \ufb02ow which is essential for context mining [17, 18]. To this end, we attempt to explore\ncontexts as early as possible and investigate architectures for action recognition.\nOur motivation also derives from the relations between CNNs and visual systems of brain since they\nshare many properties [9, 10]. One remarkable difference is that abundant recurrent connections\nexist in the visual system of brain [3] while CNNs only have forward connections. Anatomical\nevidences have shown that recurrent synapses typically outnumber feed-forward, top-down and\nfeedback synapses in the neocortex [4, 37]. This makes visual recognition tend to be a dynamic\nprocedure. Hence, we investigate to insert recurrent connections in the deployment of our architecture.\nRecent works utilize recurrent neural networks (RNNs) with long-short term memory (LSTM) units\nat the end CNN-based features of each frame to exploit semantic combinations [5, 25, 35]. These\n\n30th Conference on Neural Information Processing Systems (NIPS 2016), Barcelona, Spain.\n\n\fmethods can be regarded as inter-image context learner. In contrast, we attempt to apply recurrent\nconnections to each level of hierarchical features to aggregate their context evolutions. Similar\nefforts have demonstrated effectivity for image analysis such as object recognition and scene parsing\n[21, 17, 18]. We extend in temporal domain and study its potential for action recognition.\nThe main contribution of this paper is summarized as follows. First, we propose a deep alternative\nneural network (DANN) for action recognition. DANN stacks alternative layers consisting of\nvolumetric convolutional layer and recurrent layer. The alternative deployment is used to preserve the\ncontexts of local features as early as possible and embed their evolutions in the hierarchical feature\nlearning procedure. Second, we introduce an adaptive method to determine the temporal size of input\nvideo clip based on the density of optical \ufb02ow energy. Instead of manual choices used in most deep\narchitectures, our method utilizes adaptive input video clips preserving long range dependencies,\nwhile not breaking semantic structures. To cope with input video clips of arbitrary sizes, we develop\na volumetric pyramid pooling layer to resize the output to \ufb01xed-size before fully connected layers.\nFinally, we conduct extensive experiments and demonstrate the bene\ufb01ts of our method with early\ncontext exploration. On two challenging benchmarks HMDB51 and UCF101, we report competitive\nor superior results to the state-of-the-art.\n\n2 Deep Alternative Neural Network\n\n2.1 Adaptive Network Input\n\nThe input size of deep networks in temporal domain is often determined empirically since it is hard to\nevaluate all the choices. Previous methods often consider short intervals such as [11, 13, 27, 28] from\n1 to 16 frames. Recent work [29] argues that human actions usually span tens or hundreds of frames\nand contain characteristic patterns with long-term temporal structure. The authors use 60-frame as\nthe network input and demonstrate its advantage over 16-frame. However, it is still an ad hoc manner\nand dif\ufb01cult to favor all the action classes. We introduce an adaptive method to automatically select\nthe most discriminative video fragments using the density of optical \ufb02ow energy. We attempt to\npreserve as much as motion information and appropriate range dependencies while not breaking their\nsemantic structures in temporal domain.\n\nFigure 1: Sample video clip at every 3 frames of class \u201cgolfswing\u201d and its optical \ufb02ow energy with\nlocal minima and maxima landmarks where landmarks approximately correspond to motion change.\n\nMany evidences show that motion energy intensity induced by human activity exhibits regular\nperiodicity [33]. This signal can be approximately estimated by optical \ufb02ow computation as shown in\nFigure 1, and is particularly suitable to address our temporal estimation due to: (1) the local minima\nand maxima landmarks probably correspond to characteristic gesture and motion; (2) it is relatively\nrobust to changes in camera viewpoint. More speci\ufb01cally, we \ufb01rst compute the optical \ufb02ow \ufb01eld\n(vx, vy) for each frame I from a video Q and de\ufb01ne its \ufb02ow energy as\n(cid:107)vx(x, y), vy(x, y)(cid:107)2\n\n(cid:88)\n\n(1)\n\ne(I) =\n\n(x,y)\u2208P\n\nwhere P is the pixel level set of selected interest points. The energy of Q is then obtained as E =\n{e(I1),\u00b7\u00b7\u00b7 , e(It)}, which is further smoothed by a Gaussian \ufb01lter to suppress noise. Subsequently,\nwe locate the local minima and maxima landmarks {t} of E and for each two consecutive landmarks\ncreate a video fragment s by extracting the frames s = {It\u22121,\u00b7\u00b7\u00b7 , It}. We average the fragment\n\n2\n\n\flength of each class and illustrate the distribution in Figure 2, which indicates that using a universal\nlength can not favor all classes. To deal with the different length of video clip, we adopt the idea\nof spatial pyramid pooling (SPP) in [8] and extend to temporal domain, developing a volumetric\npyramid pooling (VPP) layer to transfer video clip of arbitrary size into a universal length in the last\nalternative layer before fully connected layer, which is presented in Section 2.3.\n\nFigure 2: Average fragment length of each class in UCF101 dataset.\n\n2.2 Alternative Layer\n\nThe key component of DANN is the alternative layer (AL), which consists of a standard volumetric\nconvolutional layer followed by a designed recurrent layer. Speci\ufb01cally, volumetric convolution is\n\ufb01rst performed to extract features from local spatiotemporal neighborhoods on feature maps in the\nprevious layers. Then a recurrent layer is applied to the output and iteratively proceeds for T times.\nThis procedure makes each unit evolve over discrete time steps and aggregate larger RFs. More\nformally, the input of a unit at position (x, y, z) in the jth feature map of the ith AL at time t, denoted\nas uxyz\n\nij (t), is given by\n\nuxyz\nij (t) = uxyz\nuxyz\nij (0) = f (wc\n\nij (0) + f (wr\n(i\u22121)juxyz\n\n(i\u22121)j)\n\nij (t \u2212 1)) + bij\n\nijuxyz\n\n(2)\n\nk and wr\n\nij (0) denotes the feed-forward output of volumetric convolutional layer, uxyz\n\nij (t \u2212 1) is the\nwhere uxyz\nrecurrent input of previous time, wc\nk are the vectorized feed-forward kernels and recurrent\nkernels, bij is the bias for jth feature map in ith layer, f is de\ufb01ned as popular recti\ufb01ed linear unit\n(ReLU) function followed by a local response normalization (LRN) [14]. The \ufb01rst term is the\noutput of volumetric convolution of previous layer and the second term is induced by the recurrent\nconnections. LRN mimics the lateral inhibition in the cortex where different features compete for\nlarge responses.\nEquation 2 describes the dynamic behavior of AL where contexts are involved after local features are\nextracted. Unfolding recurrent connection for T time steps results in a feed-forward subnetwork of\ndepth T + 1 as shown in Figure 3. While the recurrent input evolves over iterations, the feed-forward\ninput remains the same in all iterations. When t = 0 only the feed forward input is present. The\neffective RF of an AL unit in the feature maps of the previous layer expands when the iteration\nnumber increases.\n\nFigure 3: Illustrations of an alternative layer (left). The unfolding recurrent procedure is on the right.\n\nThe recurrent connections in AL provide two advantages. First, they enable every unit to incorporate\ncontexts in an arbitrarily large region in the current layer. As the time steps increase, the state of\nevery unit is in\ufb02uenced by other units in a larger and larger neighborhood in the current layer. As\n\n3\n\n\fa consequence, the size of regions that each unit can \u201cwatch\u201d in the input space also increases. In\nstandard volumetric convolutions, the size of effective RFs of the units in the current layer is \ufb01xed,\nand \u201cwatching\u201d a larger region is only possible for units in higher layers. But unfortunately the\ncontext seen by higher-level units cannot in\ufb02uence the states of the units in the current layer without\ntop-down connections. Second, the recurrent connections increase the network depth while keeping\nthe number of adjustable parameters constant by weight sharing, since AL consumes only extra\nconstant parameters of a recurrent kernel size compared with standard volumetric convolutional layer.\n\n2.3 Volumetric Pyramid Pooling Layer\n\nThe AL accepts input video clips of arbitrary sizes and produces outputs of variable sizes. However,\nthe fully connected layers require \ufb01xed-length vectors. Similar phenomenon can be found in region\nCNN (R-CNN) [6] where the input image patch is of arbitrary size. To adopt DANN for input video\nclips of arbitrary sizes, we replace the last pooling layer with a volumetric pyramid pooling layer\n(VPPL) inspired by the success of spatial pyramid pooling layer (SPPL) [8]. Figure 4 illustrates the\nstructure of VPPL. In each volumetric bin, we pool the responses of each kernel (throughout this\npaper we use max pooling). The outputs of the volumetric pyramid pooling are kM-dimensional\nvectors where M is the number of bins and k is the number of kernels in the last alternative layer.\nThe \ufb01xed-dimensional vectors are then sent to the fully connected layers.\n\nFigure 4: A network structure with volumetric pyramid pooling layer (VPPL) to resize feature maps\nof arbitrary size to \ufb01xed size.\n\nWith VPPL, the input video clips can be of any sizes. This allows not only arbitrary aspect ratios, but\nalso arbitrary scales. One can apply more compact video clips only containing semantic regions such\nas action tubes in [7] to DANN with our VPPL to pursue potential improvement.\n\n2.4 Overall Architecture\n\nFigure 5: DANN has 6 alternative layers, 5 volumetric pooling layers, 1 volumetric pyramid pooling\nlayer, 3 fully conncected layers and a softmax layer. Number of kernels are denoted in each box.\n\nOur network architecture DANN is illustrated in Figure 5. The network has 6 alternative layers with\n64, 128, 256, 256, 512 and 512 kernel response maps, followed by a volumetric pyramid pooling\nlayer and 3 fully connected layers of size 2048 each. Following [28] we use 3 \u00d7 3 \u00d7 3 kernel for\nvolumetric convolutional layer and recurrent layers of all 6 alternative layers. After each alternative\nlayer, the network includes a ReLU and a volumetric max pooling layer. Max pooling kernels are of\nsize 2\u00d72\u00d72 except in the \ufb01rst layer, where it is 2\u00d72\u00d71. All of these volumetric convolutional layers\nand recurrent layers are applied with appropriate padding and stride in both spatial and temporal\ndimensions. VPPL is applied to resize the output of the last AL to \ufb01xed-size which is the input of\nfully connected layers. Fully connected layers are followed by ReLU layers and a softmax at the end\nof the network, which outputs class scores.\n\n4\n\n\f3\n\nImplementation details\n\nThe major implementations of DANN including volumetric convolutions, recurrent layers and\noptimizations are derived from Torch toolbox platform [2].\nData Augmentation. Inspired by the random spatial cropping during training [23], we apply the\ncorresponding augmentation to spatiotemporal dimension, which we call random clipping. During\ntraining stage, given an input video, we \ufb01rst determine their temporal size t as discussed in Section\n2.1. Then we randomly select point (x, y, z) to sample a video clip of \ufb01xed size 80 \u00d7 80 \u00d7 t. A\ncommon alternative is to pre-process data by using a sliding window approach to have pre-segmented\nclips. However, this approach limits the amount of data when the windows are not overlapped as in\n[28]. Another data augmentation method that we evaluate is a multi-scale cropping similar to [32].\nTraining. We use SGD applied to mini-batches with negative log likelihood criterion. The size of\nmini-batch is set 30. Training is performed by minimizing the cross-entropy loss function using\nthe backpropagation through time (BPTT) algorithm [34]. This is equivalent to using the standard\nBP algorithm on the time-unfolded network. The \ufb01nal gradient of a shared weight is the sum of its\ngradients over all time steps. The initial learning rate for networks learned from scratch is 3 \u00d7 10\u22123\nand it is 3 \u00d7 10\u22124 for networks \ufb01ne-tuned from pre-trained models. The above schedule is used\ntogether with 0.9 dropout ratio. The momentum is set to 0.9 and weight decay is initialized with\n5 \u00d7 10\u22123 and reduced by 10\u22121 factor at every decrease of the learning rate.\nTesting. At test time, a video is also applied with temporal estimation in Section 2.1 and divided into\n80 \u00d7 80 \u00d7 t clips with a temporal stride of 4 frames, where t is the adaptive temporal size. Each clip\nis further tested with 10 crops, namely 4 corners and the center, together with their horizontal \ufb02ips.\nThe video-level score is obtained by averaging all the clip-level scores and crop scores.\n\n4 Evaluations\n\n4.1 Datasets\n\nThe evaluation is performed on UCF101 [24] and HMDB51 [15] benchmarks. Speci\ufb01cally, UCF101\ncontains 13K videos, annotated into 101 classes while HMDB51 includes 6.8K videos of 51 actions.\nThe evaluation protocol is the same for both datasets: the organisers provide three training and test\nsplits, and the performance is measured by the mean classi\ufb01cation accuracy across the splits. Each\nUCF101 split contains 9.5K training videos while HMDB51 split contains 3.7K training videos.\n\n4.2 Quantitative Results\n\nWe \ufb01rst evaluate several experimental deployment choices and determine the common settings. Then\nwe study the impact of different con\ufb01gurations of our DANN and investigate the optimal architecture.\nFinally, we report our best model and compare with state-of-the-art results.\nOptical \ufb02ow quality. We used three types of optical \ufb02ow as input signal. The performance in\ufb02uence\nis summarized in Table 1(a). We observe that sparse optical \ufb02ow consistently outperforms RGB. The\nuse of TVL1 suggested in [32] allows an almost 20% increase in performance. This demonstrates\nthat action recognition is more easy to learn from motion information compared to raw pixel values.\nGiven such results, we choose TVL1 optical \ufb02ow for all remaining experiments in this paper.\nData augmentation. Table 1(b) demonstrates the in\ufb02uence of data augmentation. Our baseline is\nsliding window with 75% overlap. On UCF101 split 1 dataset, we \ufb01nd random clipping and multi-\nscale clipping both outperform the baseline and their combination can further boost the performance.\nThus we use the combination strategy in the following experiments.\nTemporal length. Another issue we discuss is that our DANN takes video clips with adaptive\ntemporal length, which is different from most existing architectures. We examine such setting by\ncomparing 6AL_VPPL_3FC with a new architecture 6AL_3FC using \ufb01xed-size temporal length of\n16-frame, 32-frame and 64-frame, while removing VPPL. The performance gain by 6AL_VPPL_3FC\non UCF101 split 1 is approximate 4.2% as shown in Table 2(a). This result veri\ufb01es the advantages of\nour adaptive method to determine temporal length for network input.\n\n5\n\n\fTable 1: Performance comparison of different input modalities and data augmentation strategies on\nUCF101 split1.\n\n(a) Impact of optical \ufb02ow quality.\n\nInput\nRGB\nMPEG [12]\nBrox [1]\nTVL1 [36]\n\nClip-level Video-level\n\n64.4\n71.3\n76.7\n78.1\n\n64.9\n73.5\n77.2\n79.6\n\n(b) Impact of data augmentation using TVL1.\nMethod\nClip-level Video-level\nSliding window\nRandom clipping\nMulti-scale clipping\nCombined\n\n74.8\n79.6\n82.4\n82.3\n\n75.4\n78.5\n81.2\n81.6\n\nAdditional training data. We conduct experiments to see if our spatio-temporal features learned on\none dataset can help to improve the accuracy of the other one. Such additional data is already known\nto improve results in some gain [22]. The performance from scratch is 56.4% while \ufb01ne-tuning\nHMDB51 from UCF101 boosts the performance to 62.5%. Similar conclusion is demonstrated in\nTable 2(b). We conclude that one can learn generic representations with DANN like C3D [28].\n\nTable 2: Performance impact of temporal length choice and additional training data.\n\n(a) Impact of temporal length on UCF101.\nTemporal length Clip-level Video-level\n\n(b) Impact of additional training data.\nAccuracy\n\nMethod\n\n16-frame\n32-frame\n64-frame\n\nAdaptive (Ours)\n\n77.2\n77.3\n79.7\n82.8\n\n77.6\n77.2\n80.1\n83.0\n\nFrom scratch UCF\n\nFine-tuning from HMDB\n\nFrom scratch HMDB\nFine-tuning from UCF\n\n80.2\n83.7\n56.4\n62.5\n\nModel Analysis. In the following we investigate the optimal con\ufb01guration of our DANN. There are\ntwo crucial settings for DANN model. The \ufb01rst one is the AL deployment including its order and\nnumber. The other one is the unfolding time T in the recurrent layers. Table 3 shows the details of\nperformance comparison, where VC is the standard volumetric convolutional layer and B_6VC_3FC\nis a baseline composed of similar con\ufb01gurations with DANN but without ALs and adaptive input\nsize choice. The \ufb01rst column of Table 3(a) only has one AL layer and the accuracy comparison\ndemonstrates the bene\ufb01ts of exploring contexts as early as possible. The right column of Table 3(a)\nshows the performance gains as the number of AL increases, which veri\ufb01es the advantages of the\ninserted recurrent layer. Table 3(b) uses 6AL_VPP_3FC to study the impact of T and the results\nprove that larger T leads to better performance. This is perhaps due to larger contexts embedded into\nDANN which are more suitable to capture semantic information.\n\nTable 3: Performance comparison with different con\ufb01gurations of DANN on UCF101 split 1.\n\n(a) Impact of the order and the number of AL using T = 3.\n\n(b) Impact of T .\n\nArchitecture\nB_6VC_3FC\nAL_5VC_VPP_3FC\nVC_AL_4VC_VPP_3FC\n2VC_AL_3VC_VPP_3FC\n3VC_AL_2VC_VPP_3FC\n4VC_AL_VC_VPP_3FC\n5VC_AL_VPP_3FC\n\nAcc. Architecture\n80.2\n85.1\n83.3\n82.4\n82.7\n81.4\n80.9\n\n2AL_4VC_VPP_3FC\n3AL_3VC_VPP_3FC\n4AL_2VC_VPP_3FC\n5AL_VC_VPP_3FC\n6AL_VPP_3FC\n\nAcc.\n85.9\n86.7\n86.4\n87.5\n87.9\n\nArchitecture\n6AL_VPP_3FC, T = 3\n6AL_VPP_3FC, T = 4\n6AL_VPP_3FC, T = 5\n6AL_VPP_3FC, T = 6\n\nAcc.\n87.9\n88.5\n88.3\n89.0\n\nCombining spatial stream. Recent work [29] demonstrates that combining appearance information\nlearned from spatial stream can improve the performance of pure 3D CNN. We examine this issue and\ntrain a network with static RGB frames similar to [22] by inputting 256 \u00d7 256 frames and cropping\nthem randomly into 224 \u00d7 224 regions. The VGG-16 [23] network pre-trained on ImageNet is\n\ufb01ne-tuned on UCF101 and HMDB51 separately. Following good practice in [32], we apply weighted\naveraging of 0.4 and 0.6 for RGB and DANN scores, respectively. Table 4 reports the \ufb01nal results of\nour best model and its fusion with spatial stream on the three splits of both datasets.\n\n6\n\n\fComparison with the state-of-the-art. Table 4 reports the best DANN model and state-of-the-art\napproaches over three splits on UCF101 and HMDB51 datasets in terms of video-level accuracy.\nAs can be seen from Table 4, trajectory-based features are still competitive in the area of deep\nlearning, especially with the help of high-order encodings or deep architectures. Fusion strategies\noften outperform pure single deep networks. Note that all the other deep networks use a pre-de\ufb01ned\ntemporal length to generate video clip as input such as 16-frame [28] and 60-frame [29], while our\nDANN determines it in an adaptive manner. Combined with spatial stream, DANN achieves the\naccuracy of 65.9% and 91.6% on HMDB51 and UCF101, separately.\n\nTable 4: Comparison with the state-of-the-art on HMDB51 and UCF101 (over three splits).\n\nMethod\nSlow fusion [13]\nC3D [28]\nTwo-Stream(spatial) [22]\nTwo-Stream(temporal) [22]\nLTC [29]\nVery deep (temporal) [32]\nVery deep (spatial) [32]\nIDT+FV [30]\nIDT+HSV [19]\nIDT+MIFS [16]\nIDT+SFV [20]\n\nHMDB UCF\n65.4\n-\n85.2\n-\n40.5\n73.0\n83.7\n54.6\n83.3\n57.9\n87.0\n-\n87.0\n-\n57.2\n85.9 Ours\n87.9\n61.1\n89.1\n65.1\n66.8\n-\n\nFusion\n\nCNN\n\nHand\n\nMethod\nTwo-stream [22]\nCNN+deep LSTM [35]\nTDD [31]\nTDD+iDT [31]\nC3D+iDT [28]\nVery deep (two-stream) [32]\nLTC+spatial\nDANN\nDANN+spatial\n\nHMDB UCF\n88.0\n59.4\n88.6\n-\n63.2\n90.3\n91.5\n65.9\n90.4\n-\n91.4\n-\n88.6\n61.5\n63.3\n89.2\n91.6\n65.9\n\n4.3 Qualitative Analysis\n\nWe present qualitative analysis of DANN and investigate where mistakes exist. We examine the per-\nclass accuracies are computed and the difference between 6AL_VPP_3FC(T = 6) and B_6VC_3FC.\nThe class with the largest improvement when 6AL_VPP_3FC(T = 6) is used instead of B_6VC_3FC\nis \u201cbowling\u201d. This action is composed of preparing for a few seconds and then throwing a bowl. The\nadaptive temporal choice determined by DANN can aggregate more reasonable semantic structures\nwhile B_6VC_3FC has to choose temporal size manually. Figure 6 illustrates sample frames\nfrom class \u201cbowling\u201d. It is clear that DANN is more likely to leverage reasonable video clips\nas network input. On the other hand, there are also a few classes that B_6VC_3FC outperforms\n6AL_VPP_3FC(T = 6) such as \u201chaircut\u201d. We also illustrate its sample frames in Figure 6. We\nspeculate this phenomenon is partly due to the rich contexts provided by 6AL_VPP_3FC(T = 6) are\nnot \ufb01t to simple actions performed in simple background.\n\nFigure 6: Sample frames of \u201cbowling\u201d and \u201chaircut\u201d. For \u201cbowling\u201d 6AL_VPP_3FC(T = 6)\nsegments video clips with 52 frames (Figure 2) which preserves temporal semantic structures. Such\nadaptive choice performs worse than baseline for \u201chaircut\u201d, where background and action are simple.\n\n5 Conclusion and Future Work\n\nThis paper introduces a deep alternative neural network (DANN) for action recognition. DANN\nstacks alternative layers which consists of a volumetric convolutional layer and a recurrent layer.\nTo preserve motion structures in temporal domain, we present an adaptive method to determine the\ntemporal size of network input and develop a volumetric pyramid pooling layer to resize the output\nbefore fully connected layers into \ufb01xed-size vector. We demonstrate the advantages of DANN on\nHMDB51 and UCF101 benchmarks and report competitive or superior results to the state-of-the-art.\n\n7\n\n\fThere still remains some potential area of improvement. The most prominent one is the input size.\nAlthough in our model we use adaptive temporal length, the spatial size is still chosen in ad hoc\nmanner. A more compact input such as action tube [7] of arbitrary size will be studied in the future,\nwhich only contains key actor and spatiotemporal movement regions.\nAcknowledgements. The work was supported by Shenzhen Peacock Plan (20130408-183003656).\n\nReferences\n[1] Thomas Brox, Andr\u00e9s Bruhn, Nils Papenberg, and Joachim Weickert. High accuracy optical\n\n\ufb02ow estimation based on a theory for warping. In ECCV, pages 25\u201336. 2004.\n\n[2] Ronan Collobert, Koray Kavukcuoglu, and Cl\u00e9ment Farabet. Torch7: A matlab-like environment\n\nfor machine learning. In BigLearn, NIPS Workshop, number EPFL-CONF-192376, 2011.\n\n[3] Peter Dayan and Laurence F Abbott. Theoretical neuroscience, volume 806. Cambridge, MA:\n\nMIT Press, 2001.\n\n[4] Gustavo Deco and Tai Sing Lee. The role of early visual cortex in visual integration: a neural\n\nmodel of recurrent interaction. European Journal of Neuroscience, 20(4):1089\u20131100, 2004.\n\n[5] Jeffrey Donahue, Lisa Anne Hendricks, Sergio Guadarrama, Marcus Rohrbach, Subhashini\nVenugopalan, Kate Saenko, and Trevor Darrell. Long-term recurrent convolutional networks\nfor visual recognition and description. In ICCV, pages 2625\u20132634, 2015.\n\n[6] Ross Girshick, Jeff Donahue, Trevor Darrell, and Jagannath Malik. Rich feature hierarchies for\n\naccurate object detection and semantic segmentation. In CVPR, pages 580\u2013587, 2014.\n\n[7] Georgia Gkioxari and Jitendra Malik. Finding action tubes. In CVPR, pages 759\u2013768, 2015.\n\n[8] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Spatial pyramid pooling in deep\n\nconvolutional networks for visual recognition. TPAMI, 37(9):1904\u20131916, 2015.\n\n[9] David H Hubel and Torsten N Wiesel. Receptive \ufb01elds of single neurones in the cat\u2019s striate\n\ncortex. The Journal of physiology, 148(3):574\u2013591, 1959.\n\n[10] David H Hubel and Torsten N Wiesel. Receptive \ufb01elds, binocular interaction and functional\n\narchitecture in the cat\u2019s visual cortex. The Journal of physiology, 160(1):106\u2013154, 1962.\n\n[11] Shuiwang Ji, Wei Xu, Ming Yang, and Kai Yu. 3d convolutional neural networks for human\n\naction recognition. TPAMI, 35(1):221\u2013231, 2013.\n\n[12] Vadim Kantorov and Ivan Laptev. Ef\ufb01cient feature extraction, encoding and classi\ufb01cation for\n\naction recognition. In CVPR, pages 2593\u20132600, 2014.\n\n[13] Andrej Karpathy, George Toderici, Sachin Shetty, Tommy Leung, Rahul Sukthankar, and Li Fei-\nFei. Large-scale video classi\ufb01cation with convolutional neural networks. In CVPR, pages\n1725\u20131732, 2014.\n\n[14] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classi\ufb01cation with deep\n\nconvolutional neural networks. In NIPS, pages 1097\u20131105, 2012.\n\n[15] Hildegard Kuehne, Hueihan Jhuang, Est\u00edbaliz Garrote, Tomaso Poggio, and Thomas Serre.\nHmdb: a large video database for human motion recognition. In ICCV, pages 2556\u20132563, 2011.\n\n[16] Zhengzhong Lan, Ming Lin, Xuanchong Li, Alex G Hauptmann, and Bhiksha Raj. Beyond\ngaussian pyramid: Multi-skip feature stacking for action recognition. In CVPR, pages 204\u2013212,\n2015.\n\n[17] Ming Liang and Xiaolin Hu. Recurrent convolutional neural network for object recognition. In\n\nCVPR, pages 3367\u20133375, 2015.\n\n[18] Ming Liang, Xiaolin Hu, and Bo Zhang. Convolutional neural networks with intra-layer\n\nrecurrent connections for scene labeling. In NIPS, pages 937\u2013945, 2015.\n\n8\n\n\f[19] Xiaojiang Peng, Limin Wang, Xingxing Wang, and Yu Qiao. Bag of visual words and fusion\nmethods for action recognition: Comprehensive study and good practice. arXiv preprint\narXiv:1405.4506, 2014.\n\n[20] Xiaojiang Peng, Changqing Zou, Yu Qiao, and Qiang Peng. Action recognition with stacked\n\n\ufb01sher vectors. In ECCV, pages 581\u2013595. 2014.\n\n[21] Pedro Pinheiro and Ronan Collobert. Recurrent convolutional neural networks for scene labeling.\n\nIn ICML, pages 82\u201390, 2014.\n\n[22] Karen Simonyan and Andrew Zisserman. Two-stream convolutional networks for action\n\nrecognition in videos. In NIPS, pages 568\u2013576, 2014.\n\n[23] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale\n\nimage recognition. arXiv preprint arXiv:1409.1556, 2014.\n\n[24] Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human\n\nactions classes from videos in the wild. arXiv preprint arXiv:1212.0402, 2012.\n\n[25] Nitish Srivastava, Elman Mansimov, and Ruslan Salakhudinov. Unsupervised learning of video\n\nrepresentations using lstms. In ICML, pages 843\u2013852, 2015.\n\n[26] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov,\nDumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions.\nIn CVPR, pages 1\u20139, 2015.\n\n[27] Graham W Taylor, Rob Fergus, Yann LeCun, and Christoph Bregler. Convolutional learning of\n\nspatio-temporal features. In ECCV, pages 140\u2013153. 2010.\n\n[28] Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri. Learning\n\nspatiotemporal features with 3d convolutional networks. In ICCV, pages 4489\u20134497, 2015.\n\n[29] G\u00fcl Varol, Ivan Laptev, and Cordelia Schmid. Long-term temporal convolutions for action\n\nrecognition. arXiv preprint arXiv:1604.04494, 2016.\n\n[30] Heng Wang and Cordelia Schmid. Action recognition with improved trajectories. In ICCV,\n\npages 3551\u20133558, 2013.\n\n[31] Limin Wang, Yu Qiao, and Xiaoou Tang. Action recognition with trajectory-pooled deep-\n\nconvolutional descriptors. In CVPR, pages 4305\u20134314, 2015.\n\n[32] Limin Wang, Yuanjun Xiong, Zhe Wang, and Yu Qiao. Towards good practices for very deep\n\ntwo-stream convnets. arXiv preprint arXiv:1507.02159, 2015.\n\n[33] RL Waters and JM Morris. Electrical activity of muscles of the trunk during walking. Journal\n\nof anatomy, 111(Pt 2):191, 1972.\n\n[34] Paul J Werbos. Backpropagation through time: what it does and how to do it. Proceedings of\n\nthe IEEE, 78(10):1550\u20131560, 1990.\n\n[35] Joe Yue-Hei Ng, Matthew Hausknecht, Sudheendra Vijayanarasimhan, Oriol Vinyals, Rajat\nMonga, and George Toderici. Beyond short snippets: Deep networks for video classi\ufb01cation. In\nCVPR, pages 4694\u20134702, 2015.\n\n[36] Christopher Zach, Thomas Pock, and Horst Bischof. A duality based approach for realtime tv-l\n\n1 optical \ufb02ow. In Pattern Recognition, pages 214\u2013223. 2007.\n\n[37] Matthew D Zeiler and Rob Fergus. Stochastic pooling for regularization of deep convolutional\n\nneural networks. arXiv preprint arXiv:1301.3557, 2013.\n\n9\n\n\f", "award": [], "sourceid": 480, "authors": [{"given_name": "Jinzhuo", "family_name": "Wang", "institution": "PKU"}, {"given_name": "Wenmin", "family_name": "Wang", "institution": "peking university"}, {"given_name": "xiongtao", "family_name": "Chen", "institution": "peking university"}, {"given_name": "Ronggang", "family_name": "Wang", "institution": "peking university"}, {"given_name": "Wen", "family_name": "Gao", "institution": "peking university"}]}