{"title": "Predicting Scene Parsing and Motion Dynamics in the Future", "book": "Advances in Neural Information Processing Systems", "page_first": 6915, "page_last": 6924, "abstract": "It is important for intelligent systems, e.g. autonomous vehicles and robotics to anticipate the future in order to plan early and make decisions accordingly. Predicting the future scene parsing and motion dynamics helps the agents better understand the visual environment better as the former provides dense semantic segmentations, i.e. what objects will be present and where they will appear, while the latter provides dense motion information, i.e. how the objects move in the future. In this paper, we propose a novel model to predict the scene parsing and motion dynamics in unobserved future video frames simultaneously. Using history information (preceding frames and corresponding scene parsing results) as input, our model is able to predict the scene parsing and motion for arbitrary time steps ahead. More importantly, our model is superior compared to other methods that predict parsing and motion separately, as the complementary relationship between the two tasks are fully utilized in our model through joint learning. To our best knowledge, this is the first attempt in jointly predicting scene parsing and motion dynamics in the future frames. On the large-scale Cityscapes dataset, it is demonstrated that our model produces significantly better parsing and motion prediction results compared to well established baselines. In addition, we also show our model can be used to predict the steering angle of the vehicles, which further verifies the ability of our model to learn underlying latent parameters.", "full_text": "Predicting Scene Parsing and Motion Dynamics\n\nin the Future\n\nXiaojie Jin1, Huaxin Xiao2, Xiaohui Shen3, Jimei Yang3, Zhe Lin3\n\nYunpeng Chen2, Zequn Jie4, Jiashi Feng2, Shuicheng Yan5,2\n\n1NUS Graduate School for Integrative Science and Engineering (NGS), NUS\n\n2Department of ECE, NUS\n\n3Adobe Research\n\n4Tencent AI Lab\n\n5Qihoo 360 AI Institute\n\nAbstract\n\nThe ability of predicting the future is important for intelligent systems, e.g. au-\ntonomous vehicles and robots to plan early and make decisions accordingly. Future\nscene parsing and optical \ufb02ow estimation are two key tasks that help agents better\nunderstand their environments as the former provides dense semantic information,\ni.e. what objects will be present and where they will appear, while the latter pro-\nvides dense motion information, i.e. how the objects will move. In this paper, we\npropose a novel model to simultaneously predict scene parsing and optical \ufb02ow in\nunobserved future video frames. To our best knowledge, this is the \ufb01rst attempt in\njointly predicting scene parsing and motion dynamics. In particular, scene parsing\nenables structured motion prediction by decomposing optical \ufb02ow into different\ngroups while optical \ufb02ow estimation brings reliable pixel-wise correspondence\nto scene parsing. By exploiting this mutually bene\ufb01cial relationship, our model\nshows signi\ufb01cantly better parsing and motion prediction results when compared\nto well-established baselines and individual prediction models on the large-scale\nCityscapes dataset. In addition, we also demonstrate that our model can be used to\npredict the steering angle of the vehicles, which further veri\ufb01es the ability of our\nmodel to learn latent representations of scene dynamics.\n\n1\n\nIntroduction\n\nFuture prediction is an important problem for arti\ufb01cial intelligence. To enable intelligent systems like\nautonomous vehicles and robots to react to their environments, it is necessary to endow them with the\nability of predicting what will happen in the near future and plan accordingly, which still remains an\nopen challenge for modern arti\ufb01cial vision systems.\nIn a practical visual navigation system, scene parsing and dense motion estimation are two essential\ncomponents for understanding the scene environment. The former provides pixel-wise prediction\nof semantic categories (thus the system understands what and where the objects are) and the latter\ndescribes dense motion trajectories (thus the system learns how the objects move). The visual\nsystem becomes \u201csmarter\u201d by leveraging the prediction of these two types of information, e.g.\npredicting how the car coming from the opposite direction moves to plan the path ahead of time\nand predict/control the steering angle of the vehicle. Despite numerous models have been proposed\non scene parsing [4, 7, 17, 26, 28, 30, 15] and motion estimation [2, 9, 21], most of them focus on\nprocessing observed images, rather than predicting in unobserved future scenes. Recently, a few\nworks [22, 16, 3] explore how to anticipate the scene parsing or motion dynamics, but they all tackle\nthese two tasks separately and fail to utilize the bene\ufb01ts that one task brings to the other.\nIn this paper, we try to close this research gap by presenting a novel model for jointly predicting scene\nparsing and motion dynamics (in terms of the dense optical \ufb02ow) for future frames. More importantly,\nwe leverage one task as the auxiliary of the other in a mutually boosting way. See Figure 1 for\n\n31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA.\n\n\fFigure 1: Our task. The proposed model jointly predicts scene parsing and optical \ufb02ow in the future. Top: Future\n\ufb02ow (highlighted in red) anticipated using preceding frames. Bottom: Future scene parsing (highlighted in red)\nanticipated using preceding scene parsing results. We use the \ufb02ow \ufb01eld color coding from [2].\n\nan illustration of our task. For the task of predictive scene parsing, we use the discriminative and\ntemporally consistent features learned in motion prediction to produce parsing prediction with more\n\ufb01ne details. For the motion prediction task, we utilize the semantic segmentations produced by\npredictive parsing to separately estimate motion for pixels with different categories. In order to\nperform the results for multiple time steps, we take the predictions as input and iterate the model\nto predict subsequent frames. The proposed model has a generic framework which is agnostic to\nbackbone deep networks and can be conveniently trained in an end-to-end manner.\nTaking Cityscapes [5] as testbed, we conduct extensive experiments to verify the effectiveness of\nour model in future prediction. Our model signi\ufb01cantly improves mIoU of parsing predictions and\nreduces the endpoint error (EPE) of \ufb02ow predictions compared to strongly competitive baselines\nincluding a warping method based on optical \ufb02ow, standalone parsing prediction or \ufb02ow prediction\nand other state-of-the-arts methods [22]. We also present how to predict steering angles using the\nproposed model.\n\n2 Related work\n\nFor the general \ufb01eld of classic \ufb02ow (motion) estimation and image semantic segmentation, which is\nout of this paper\u2019s scope, we refer the readers to comprehensive review articles [2, 10]. Below we\nmainly review existing works that focus on predictive tasks.\n\nFlow and scene parsing prediction The research on predictive scene parsing or motion prediction\nis still relatively under-explored. All existing works in this direction tackle the parsing prediction and\n\ufb02ow prediction as independent tasks. With regards to motion prediction, Luo et al. [19] employed a\nconvolutional LSTM architecture to predict sequences of 3D optical \ufb02ow. Walker et al. [35] made\nlong-term motion and appearance prediction via a transition and context model. [31] trained CNN for\npredicting motion of handwritten characters in a synthetic dataset. [36] predicted future optical \ufb02ow\ngiven a static image. Different from above works, our model not only predicts the \ufb02ow but also scene\nparsing at the same time, which de\ufb01nitely provides richer information to visual systems.\nThere are also only a handful number of works exploring the prediction of scene parsing in future\nframes. Jin et al. [16] trained a deep model to predict the segmentations of the next frame from\npreceding input frames, which is shown to be bene\ufb01cial for still-image parsing task. Based on the\nnetwork proposed in [20], Natalia et al. [22] predicted longer-term parsing maps for future frames\nusing the preceding frames\u2019 parsing maps. Different from [22], we simultaneously predict optical\n\ufb02ows for future frames. Bene\ufb01ted from the discriminative local features learned from \ufb02ow prediction,\nthe model produces more accurate parsing results. Another related work to ours is [24] which\nemployed an RNN to predict the optical \ufb02ow and used the \ufb02ow to warp preceding segmentations.\nRather than simply producing the future parsing map through warping, our model predicts \ufb02ow and\nscene parsing jointly using learning methods. More importantly, we leverage the bene\ufb01t that each\ntask brings to the other to produce better results for both \ufb02ow prediction and parsing prediction.\n\nPredictive learning While there are few works speci\ufb01cally on predictive scene parsing or dense\nmotion prediction, learning to prediction in general has received a signi\ufb01cant attention from the\n\n2\n\n4\uf02dtXInput3\uf02dtXInput2\uf02dtXInput1\uf02dtXInputt\uf04fOutputtSOutput1\uf02dtSInput2\uf02dtSInput3\uf02dtSInput4\uf02dtSInput\u2026\u2026dt\uf02b\uf04fOutputdtS\uf02bOutput\fFigure 2: The framework of our model for predicting future scene parsing and optical \ufb02ow for one time step\nahead. Our model is motivated by the assumption that \ufb02ow and parsing prediction are mutually bene\ufb01cial. We\ndesign the architecture to promote such mutual bene\ufb01ts. The model consists of two module networks, i.e. the\n\ufb02ow anticipating network (blue) which takes preceding frames: Xt\u22124:t\u22121 as input and predicts future \ufb02ow and\nthe parsing anticipating network (yellow) which takes the preceding parsing results: St\u22124:t\u22121 as input and\npredicts future scene parsing. By providing pixel-level class information (i.e. St\u22121), the parsing anticipating\nnetwork bene\ufb01ts the \ufb02ow anticipating network to enable the latter to semantically distinguish different pixels\n(i.e. moving/static/other objects) and predict their \ufb02ows more accurately in the corresponding branch. Through\nthe transform layer, the discriminative local features learned by the \ufb02ow anticipating network are combined\nwith the parsing anticipating network to facilitate parsing over small objects and avoid over-smooth in parsing\npredictions. When predicting multiple time-steps ahead, the prediction of the parsing network in a time-step is\nused as the input in the next time-step.\n\nresearch community in recent years. Research in this area has explored different aspects of this\nproblem. [37] focused on predicting the trajectory of objects given input image. [13] predicted\nthe action class in the future frames. Generative adversarial networks (GAN) are \ufb01rstly introduced\nin [11] to generate natural images from random noise, and have been widely used in many \ufb01elds\nincluding image synthesis [11], future prediction [18, 20, 34, 36, 32, 33] and semantic inpainting [23].\nDifferent from above methods, our model explores a new predictive task, i.e. predicting the scene\nparsing and motion dynamics in the future simultaneously.\n\nMulti-task learning Multi-task learning [1, 6] aims to solve multiple tasks jointly by taking\nadvantage of the shared domain knowledge in related tasks. Our work is partially related to multi-task\nlearning in that both the parsing results and motion dynamics are predicted jointly in a single model.\nHowever, we note that predicting parsing and motion \u201cin the future\u201d is a novel and challenging task\nwhich cannot be straightforwardly tackled by conventional multi-task learning methods. To our best\nknowledge, our work is the \ufb01rst solution to this challenging task.\n\n3 Predicting scene parsing and motion dynamics in the future\n\nIn this section, we \ufb01rst propose our model for predicting semantics and motion dynamics one time\nstep ahead, and then extend our model to perform predictions for multiple time steps.\nDue to high cost of acquiring dense human annotations of optical \ufb02ow and scene parsing for\nnatural scene videos, only subset of frames are labeled for scene parsing in the current datasets.\nFollowing [22], to circumvent the need for datasets with dense annotations, we train an adapted\nRes101 model (denoted as Res101-FCN, more details are given in Sec. 4.1) for scene parsing to\nproduce the target semantic segmentations for frames without human annotations. Similarly, to obtain\nthe dense \ufb02ow map for each frame, we use the output of the state-of-the-art epic\ufb02ow [25] as our target\noptical \ufb02ow. Note that our model is orthogonal to speci\ufb01c \ufb02ow methods since they are only used to\nproduce the target \ufb02ow for training the \ufb02ow anticipating network. Notations used in the following text\nare as follows. Xi denotes the i-th frame of a video and Xt\u2212k:t\u22121 denotes the sequence of frames\nwith length k from Xt\u2212k to Xt\u22121. The semantic segmentation of Xt is denoted as St, which is the\n\n3\n\n4\uf02dtX1\uf02dtX\u2026CNN12 Res. BlocksUp-samplingConvTransformLayer4\uf02dtS1\uf02dtS\u2026CNN2OBJ-OTHflowLsegLOBJ-STAflowLOBJ-MOVflowLFlow Anticipating NetworkParsing Anticipating Network\foutput of the penultimate layer of Res101-FCN. St has the same spatial size as Xt and is a vector of\nlength C at each location, where C is the number of semantic classes. We denote Ot as the pixel-wise\noptical \ufb02ow map from Xt\u22121 to Xt, which is estimated via epic\ufb02ow [25]. Correspondingly, \u02c6St and\n\u02c6Ot denote the predicted semantic segmentation and optical \ufb02ow.\n\n3.1 Prediction for one time step ahead\n\nModel overview The key idea of our approach is to model \ufb02ow prediction and parsing prediction\njointly, which are potentially mutually bene\ufb01cial. As illustrated in Figure 2, the proposed model\nconsists of two module networks that are trained jointly, i.e. the \ufb02ow anticipating network that takes\npreceding frames Xt\u2212k:t\u22121 as input to output the pixelwise \ufb02ow prediction for Ot (from Xt\u22121 to\nXt), and the parsing anticipating network that takes the segmentation of preceding frames St\u2212k:t\u22121\nas input to output pixelwise semantic prediction for an unobserved frame Xt. The mutual in\ufb02uences\nof each network on the other are exploited in two aspects. First, the last segmentations St\u22121 produced\nby the parsing anticipating network convey pixel-wise class labels, which are used by the \ufb02ow\nanticipating network to predict optical \ufb02ow values for each pixel according to its belonging object\ngroup, e.g. moving objects or static objects. Second, the parsing anticipating network combines the\ndiscriminative local feature learned by the \ufb02ow anticipating network to produce sharper and more\naccurate parsing predictions.\nSince both parsing prediction and \ufb02ow prediction are essentially both the dense classi\ufb01cation problem,\nwe use the same deep architecture (Res101-FCN) for predicting parsing results and optical \ufb02ow. Note\nthe Res101-FCN used in this paper can be replaced by any CNNs. We adjust the input/output layers\nof these two networks according to the different channels of their input/output. The features extracted\nby feature encoders (CNN1 and CNN2) are spatially enlarged via up-sampling layers and \ufb01nally fed\nto a convolutional layer to produce pixel-wise predictions which have the same spatial size as input.\n\nFlow anticipating network In videos captured for autonomous driving or navigation, regions\nwith different class labels have different motion patterns. For example, the motion of static\nobjects like road is only caused by the motion of the camera while the motion of moving ob-\njects is a combination of motions from both the camera and objects themselves. Therefore\ncompared to methods that predict all pixels\u2019 optical \ufb02ow in a single output layer, it would\nlargely reduce the dif\ufb01culty of feature learning by separately modeling the motion of regions\nwith different classes. Following [29], we assign each class into one of three pre-de\ufb01ned object\ngroups, i.e. G = {moving objects (MOV-OBJ), static objects (STA-OBJ), other objects (OTH-OBJ)}\nin which MOV-OBJ includes pedestrians, truck, etc., STA-OBJ includes sky, road, etc., and OTH-OBJ\nincludes vegetation and buildings, etc. which have diverse motion patterns and shapes. We append a\nsmall network (consisting of two residual blocks) to the feature encoder (CNN1) for each object group\nto learn speci\ufb01ed motion representations. During training, the loss for each pixel is only generated at\nthe branch that corresponds to the object group to which the pixel belongs. Similarly, in testing, the\n\ufb02ow prediction for each pixel is generated by the corresponding branch. The loss function between\nthe model output \u02c6Ot and target output Ot is\n\n(cid:88)\n\ng\u2208G\n\nL\ufb02ow( \u02c6Ot, Ot) =\n\nLg\n\ufb02ow; Lg\n\n\ufb02ow =\n\n1\n|Ng|\n\n(i,j)\u2208Ng\n\n(1)\n\n(cid:88)\n\n(cid:13)(cid:13)(cid:13)Oi,j\n\nt \u2212 \u02c6Oi,j\n\nt\n\n(cid:13)(cid:13)(cid:13)2\n\nwhere (i, j) index the pixel in the region Ng.\n\nParsing anticipating network The input of the parsing anticipating network is a sequence of\npreceding segmentations St\u2212k:t\u22121. We also explore other input space alternatives, including preced-\ning frames Xt\u2212k:t\u22121, and the combination of preceding frames and corresponding segmentations\nXt\u2212k:t\u22121St\u2212k:t\u22121, and we observe that the input St\u2212k:t\u22121 achieves the best prediction performance.\nWe conjecture it is easier to learn the mapping between variables in the same domain (i.e. both\nare semantic segmentations). However, there are two drawbacks brought by this strategy. Firstly,\nSt\u2212k:t\u22121 lose the discriminative local features e.g. color, texture and shape etc., leading to the missing\nof small objects in predictions, as illustrated in Figure 3 (see yellow boxes). The \ufb02ow prediction\nnetwork may learn such features from the input frames. Secondly, due to the lack of local features\nin St\u2212k:t\u22121, it is dif\ufb01cult to learn accurate pixel-wise correspondence in the parsing anticipating\n\n4\n\n\fnetwork, which causes the predicted labeling maps to be over-smooth, as shown in Figure 3. The\n\ufb02ow prediction network can provide reliable dense pixel-wise correspondence by regressing to the\ntarget optical \ufb02ow. Therefore, we integrate the features learned by the \ufb02ow anticipating network with\nthe parsing prediction network through a transform layer (a shallow CNN) to improve the quality of\npredicted labeling maps. Depending on whether human annotations are available, the loss function is\nde\ufb01ned as\n\nLseg( \u02c6S, S) =\n\n(i,j)\u2208Xt\n\nL(cid:96)1 ( \u02c6S, S) + Lgdl( \u02c6S, S), otherwise\n\nlog( \u02c6Si,j\n\nt (c)), Xt has human annotation,\n\n(2)\n\nwhere c is the ground truth class for the pixel at location (i, j). It is a conventional pixel-wise\ncross-entropy loss when Xt has human annotations. L(cid:96)1 and Lgdl are (cid:96)1 loss and gradient difference\nloss [20] which are de\ufb01ned as\n\n\uf8f1\uf8f2\uf8f3\u2212 (cid:80)\n(cid:12)(cid:12)(cid:12),\n\nt \u2212 \u02c6Si,j\n\nt\n\n(cid:12)(cid:12)(cid:12)Si,j\n(cid:16)(cid:12)(cid:12)(cid:12)|Si,j\n\nt \u2212 Si\u22121,j\n\nt\n\nL(cid:96)1 ( \u02c6S, S) =\n\nLgdl =\n\n(cid:88)\n(cid:88)\n\n(i,j)\u2208Xt\n\n(i,j)\u2208Xt\n\n| \u2212 | \u02c6Si,j\n\nt \u2212 \u02c6Si\u22121,j\n\nt\n\n|(cid:12)(cid:12)(cid:12) +\n\n(cid:12)(cid:12)(cid:12)|Si,j\u22121\n\nt\n\n\u2212 Si,j\n\nt\n\n| \u2212 | \u02c6Si,j\u22121\n\nt\n\n\u2212 \u02c6Si,j\n\nt\n\n|(cid:12)(cid:12)(cid:12)(cid:17)\n\n.\n\nThe (cid:96)1 loss encourages predictions to regress to the target values while the gradient difference loss\nproduces large errors in the gradients of the target and predictions.\nThe reason for using different losses for human and non-human annotated frames in Eq. 2 is that\nthe automatically produced parsing ground-truth (by the pre-trained Res101-FCN) of the latter may\ncontain wrong annotations. The cross-entropy loss using one-hot vectors as labels is sensitive to the\nwrong annotations. Comparatively, the ground-truth labels used in the combined loss (L(cid:96)1 + Lgdl) are\ninputs of the softmax layer (ref. Sec. 3) which allow for non-zero values in more than one category,\nthus our model can learn useful information from the correct category even if the annotation is wrong.\nWe \ufb01nd replacing L(cid:96)1 + Lgdl with the cross-entropy loss reduces the mIoU of the baseline S2S (i.e.\nthe parsing participating network) by 1.5 from 66.1 when predicting the results one time-step ahead.\nNow we proceed to explain the role of the transform layer which transforms the features of CNN1\nbefore combining them with those of CNN2. Compared with naively combining the features from two\nnetworks (e.g., concatenation), the transform layer brings the following two advantages: 1) naturally\nnormalize the feature maps to proper scales; 2) align the features of semantic meaning such that the\nintegrated features are more powerful for parsing prediction. Effectiveness of this transform layer is\nclearly validated in the ablation study in Sec. 4.2.1.\nThe \ufb01nal objective of our model is to minimize the combination of losses from the \ufb02ow anticipating\nnetwork and the parsing anticipating network as follows\n\nL(Xt\u2212k:t\u22121, St\u2212k:t\u22121, \u02c6Xt, \u02c6St) = L\ufb02ow( \u02c6Ot, Ot) + Lseg( \u02c6S, S).\n\n3.2 Prediction for multiple time steps ahead\n\nBased on the above model which predicts scene parsing and \ufb02ow for the single future time step, we\nexplore two ways to predict further into the future. Firstly, we iteratively apply the model to predict\none more time step into the future by treating the prediction as input in a recursive way. Speci\ufb01cally,\nfor predicting multiple time steps in the \ufb02ow anticipating network, we warp the most recent frame\nXt\u22121 using the output prediction \u02c6Ot to get the \u02c6Xt which is then combined with Xt\u2212k\u22121:t\u22121 to feed\nthe \ufb02ow anticipating network to generate \u02c6Ot+1, and so forth. For the parsing anticipating network, we\ncombine the predicted parsing map \u02c6St with St\u2212k\u22121:t\u22121 as the input to generate the parsing prediction\nat t + 1. This scheme is easy to implement and allows us to predict arbitrarily far into the future\nwithout increasing training complexity w.r.t. with the number of time-steps we want to predict.\nSecondly, we \ufb01ne-tune our model by taking into account the in\ufb02uence that the recurrence has on\nprediction for multiple time steps. We apply our model recurrently as described above to predict two\ntime steps ahead and apply the back propagation through time (BPTT) [14] to update the weight. We\nhave veri\ufb01ed through experiments that the \ufb01ne-tuning approach can further improve the performance\nas it models longer temporal dynamics during training.\n\n5\n\n\fFigure 3: Two examples of prediction results for predicting one time step ahead. Odd row: The images from\nleft to right are Xt\u22122, Xt\u22121, the target optical \ufb02ow map Ot, the \ufb02ow predictions from PredFlow and the \ufb02ow\npredictions from our model. Even row: The images from left to right are St\u22122, St\u22121, the ground truth semantic\nannotations at the time t, the parsing prediction from S2S and the parsing prediction from our model. The \ufb02ow\npredictions from our model show clearer object boundaries and predict more accurate values for moving objects\n(see black boxes) compared to PredFlow. Our model is superior to S2S by being more discriminative to the\nsmall objects in parsing predictions (see yellow boxes).\n\nFigure 4: An example of prediction results for predicting ten time steps ahead. Top (from left to right): Xt\u221211,\nXt\u221210, the target optical \ufb02ow map Ot, the \ufb02ow prediction from PredFlow and the \ufb02ow prediction from our\nmodel. Bottom (from left to right): St\u221211, St\u221210, the ground truth semantic annotation at the time t, the parsing\nprediction from S2S and the parsing prediction from our model. Our model outputs better prediction compared\nto PredFlow (see black boxes) and S2S (see yellow boxes).\n\n4 Experiment\n\n4.1 Experimental settings\n\nDatasets We verify our model on the large scale Cityscapes [5] dataset which contains 2,975/500\ntrain/val video sequences with 19 semantic classes. Each video sequence lasts for 1.8s and contains\n30 frames, among which the 20th frame has \ufb01ne human annotations. Every frame in Cityscapes has a\nresolution of 1,024 \u00d7 2,048 pixels.\n\nEvaluation criteria We use the mean IoU (mIoU) for evaluating the performance of predicted\nparsing results on those 500 frames in the val set with human annotations. For evaluating the\nperformance of \ufb02ow prediction, we use the average endpoint error (EPE) [2] following conventions [8]\nwhich is de\ufb01ned as 1\nN\nand v are the components of optical \ufb02ow along x and y directions, respectively. To be consistent with\nmIoU, EPEs are also reported on the 20th frame in each val sequence.\n\n(cid:112)(u \u2212 uGT)2 + (v \u2212 vGT)2 where N is the number of pixels per-frame, and u\n\nBaselines To fully demonstrate the advantages of our model on producing better predictions, we\ncompare our model against the following baseline methods:\n\n6\n\n\fTable 1: The performance of parsing prediction on\nCityscapes val set. For each competing model, we list\nthe mIoU/EPE when predicting one time step ahead.\nBest results in bold.\n\nModel\nCopy last input\nWarp last input\nPredFlow\nS2S [22]\nours (w/o Trans. layer)\nours\n\nmIoU EPE\n3.03\n59.7\n3.03\n61.3\n61.3\n2.71\n62.6\n64.7\n66.1\n\n2.42\n2.30\n\n-\n\nTable 2: The performance of motion prediction on\nCityscapes val set. For each model, we list the\nmIoU/EPE when predicting one time step ahead. Best\nresults in bold.\nModel\nCopy last input\nWarp last input\nPredFlow\nS2S [22]\nours (w/o Recur. FT)\nours\n\nmIoU EPE\n9.40\n41.3\n9.40\n42.0\n43.6\n8.10\n50.8\n52.6\n53.9\n\n6.63\n6.31\n\n-\n\n\u2022 Copy last input Copy the last optical \ufb02ow (Ot\u22121) and parsing map (St\u22121) at time t \u2212 1 as\npredictions at time t.\n\n\u2022 Warp last input Warp the last segmentation St\u22121 using Ot\u22121 to get the parsing prediction\nat the next time step. In order to make \ufb02ow applicable to the correct locations, we also warp\nthe \ufb02ow \ufb01eld using the optical \ufb02ow in each time step.\n\n\u2022 PredFlow Perform \ufb02ow prediction without the object masks generated from segmentations.\nThe architecture is the same as the \ufb02ow prediction net in Figure 2 which generates pixel-wise\n\ufb02ow prediction in a single layer, instead of multiple branches. For fair comparison with our\njoint model, in the following we report the average result of two independent PredFlow with\ndifferent random initializations. When predicting the segmentations at time t, we use the\n\ufb02ow prediction output by PredFlow at time t to warp the segmentations at time t \u2212 1. This\nbaseline aims to verify the advantages brought by parsing prediction when predicting \ufb02ow.\n\u2022 S2S [22] Use only parsing anticipating network. The difference is that the former does not\nleverage features learned by the \ufb02ow anticipating network to produce parsing predictions.\nWe replace the backbone network in the original S2S as the same one of ours, i.e. Res101-\nFCN and retrain S2S with the same con\ufb01gurations as those of ours. Similar to the PredFlow,\nthe average performance of two randomly initialized S2S is reported. This baseline aims to\nverify the advantages brought by \ufb02ow prediction when predicting parsing.\n\nImplementation details Throughout the experiments, we set the length of the input sequence\nas 4 frames, i.e. k = 4 in Xt\u2212k:t\u22121 and St\u2212k:t\u22121 (ref. Sec. 3). The original frames are \ufb01rstly\ndownsampled to the resolution of 256 \u00d7 512 to accelerate training. In the \ufb02ow anticipating network,\nwe assign 19 semantic classes into three object groups which are de\ufb01ned as follows: MOV-OBJ\nincluding person, rider, car, truck, bus, train, motorcycle and bicycle, STA-OBJ including road,\nsidewalk, sky, pole, traf\ufb01c light and traf\ufb01c sign and OTH-OBJ including building, wall, fence, terrain\nand vegetation. For data augmentation, we randomly crop a patch with the size of 256 \u00d7 256 and\nperform random mirror for all networks. All results of our model are based on single-model single-\nscale testing. For other hyperparameters including weight decay, learning rate, batch size and epoch\nnumber etc., please refer to the supplementary material. All of our experiments are carried out on\nNVIDIA Titan X GPUs using the Caffe library.\n\n4.2 Results and analysis\n\nExamples of the \ufb02ow predictions and parsing predictions output by our model for one-time step and\nten-time steps are illustrated in Figure 3 and Figure 4 respectively. Compared to baseline models, our\nmodel produces more visually convincing prediction results.\n\n4.2.1 One-time step anticipation\n\nTable 1 lists the performance of parsing and \ufb02ow prediction on the 20th frame in the val set which has\nground truth semantic annotations. It can be observed that our model achieves the best performance on\nboth tasks, demonstrating the effectiveness on learning the latent representations for future prediction.\nBased on the results, we analyze the effect of each component in our model as follows.\n\n7\n\n\fThe effect of \ufb02ow prediction on parsing prediction Compared with S2S which does not leverage\n\ufb02ow predictions, our model improves the mIoU with a large margin (3.5%). As shown in Figure 3,\ncompared to S2S, our model performs better on localizing the small objects in the predictions e.g.\npedestrian and traf\ufb01c sign, because it combines the discriminative local features learned in the \ufb02ow\nanticipating network. These results clearly demonstrate the bene\ufb01t of \ufb02ow prediction for parsing\nprediction.\nThe effect of parsing prediction on \ufb02ow prediction Compared with the baseline PredFlow which\nhas no access to the semantic information when predicting the \ufb02ow, our model reduces the average\nEPE from 2.71 to 2.30 (a 15% improvement), which demonstrates parsing prediction is bene\ufb01cial to\n\ufb02ow prediction. As illustrated in Figure 3, the improvement our model makes upon PredFlow comes\nfrom two aspects. First, since the segmentations provide boundary information of objects, the \ufb02ow\nmap predicted by our model has clearer object boundaries while the \ufb02ow map predicted by PredFlow\nis mostly blurry. Second, our model shows more accurate \ufb02ow predictions on the moving objects\n(ref. Sec. 4.1 for the list of moving objects). We calculate the average EPE for only the moving\nobjects, which is 2.45 for our model and 3.06 for PredFlow. By modeling the motion of different\nobjects separately, our model learns better representation for each motion mode. If all motions are\npredicted in one layer as in PredFlow, then the moving objects which have large displacement than\nother regions are prone to smoothness.\nBene\ufb01ts of the transform layer As introduced in Sec. 3.1, the transform layer improves the perfor-\nmance of our model by learning the latent feature space transformations from CNN1 to CNN2. In our\nexperiments, the transform layer contains one residual block [12] which has been widely used due to\nits good performance and easy optimization. Details of the residual block used in our experiments\nare included in the supplementary material. Compared to the variant of our model w/o the transform\nlayer, adding the transform layer improves the mIoU by 1.4 and reduces EPE by 0.12. We observe\nthat stacking more residual blocks only leads to marginal improvements at larger computational costs.\n\n4.2.2 Longer duration prediction\n\nThe comparison of the prediction performance among all methods for ten time steps ahead is listed in\nTable 2, from which one can observe that our model performs the best in this challenging task. The\neffect of each component in our model is also veri\ufb01ed in this experiment. Speci\ufb01cally, compared with\nS2S, our model improves the mIoU by 3.1% due to the synergy with the \ufb02ow anticipating network.\nThe parsing prediction helps reducing the EPE of PredFlow by 1.79. Qualitative results are illustrated\nin Figure 4.\nThe effect of recurrent \ufb01ne-tuning As explained in Sec. 3.2, it helps our model to capture long term\nvideo dynamics by \ufb01ne-tuning the weights when recurrently applying the model to predict the next\ntime step in the future. As shown in Table 2, compared to the variant w/o recurrent ft, our model w/\nrecurrent \ufb01ne-tuning improves the mIoU by 1.3% and reduces the EPE by 0.32, therefore verifying\nthe effect of recurrent \ufb01ne-tuning.\n\n4.3 Application for predicting the steering angle of a vehicle\n\nWith the parsing prediction and \ufb02ow prediction\navailable, one can enable the moving agent to be\nmore alert about the environments and get \u201csmarter\u201d.\nHere, we investigate one application: predicting the\nsteering angle of the vehicle. The intuition is it is\nconvenient to infer the steering angle given the pre-\ndicted \ufb02ow of static objects, e.g. road and sky, the\nmotion of which is only caused by ego-motion of\nthe camera mounted on the vehicle. Speci\ufb01cally, we\nappend a fully connected layer to take the features\nlearned in the STA-OBJ branch in the \ufb02ow antici-\npating network as input and perform regression to\nsteering angles. We test our model on the dataset\nfrom Comma.ai [27] which consists of 11 videos\n\n1https://github.com/commaai/research\n\n8\n\nTable 3: Comparison results of steering angle predic-\ntion on a dataset from Comma.ai [27]. The criteria\nis the mean square error (MSE, in degree2) between\nthe prediction and groud truth.\n\nModel\nCopy last prediction\nComma.ai1 [27]\nours\n\nMSE (degrees2)\n\n4.81\n\u223c 4\n2.96\n\n\famounting to about 7 hours. The data of steering angles have been recorded for each frame captured\nat 20Hz with the resolution of 160 \u00d7 320. We randomly sample 50K/5K frames from the train set for\ntraining and validation purpose. Since there are videos captured at night, we normalize all training\nframes to [0, 255]. Similar to Cityscapes, we use epic\ufb02ow and Res101-FCN to produce the target\noutput for \ufb02ow prediction and parsing prediction, respectively. We \ufb01rst train our model following\nSec. 3 and then \ufb01ne-tune the whole model with the MSE loss after adding the fully connected layer\nfor steering angle prediction. During training, random crop with the size of 160 \u00d7 160 and random\nmirror are employed and other hyperparameter settings follow Sec. 4.1. The testing results are listed\nin Table 3. Compared to the model from Comma.ai which uses a \ufb01ve-layer CNN to estimate the\nsteering angle from a single frame and is trained end-to-end on all the training frames (396K), our\nmodel achieves much better performance (2.84 versus \u223c4 in degrees2). Although we do not push the\nperformance by using more training data and more complex prediction models (only a fully connected\nlayer is used in our model for output steering angle), this preliminary experiment still veri\ufb01es the\nadvantage of our model in learning the underlying latent parameters. We think it is just an initial\nattempt in validating the dense prediction results through applications, which hopefully can stimulate\nresearchers to explore other interesting ways to utilize the parsing prediction and \ufb02ow prediction.\n\n5 Conclusion\n\nIn this paper, we proposed a novel model to predict the future scene parsing and motion dynamics.\nTo our best knowledge, this is the \ufb01rst research attempt to anticipate visual dynamics for building\nintelligent agents. The model consists of two networks: the \ufb02ow anticipating network and the\nparsing anticipating network which are jointly trained and bene\ufb01t each other. On the large scale\nCityscapes dataset, the experimental results demonstrate that the proposed model generates more\naccurate prediction than well-established baselines both on single time step prediction and multiple\ntime prediction. In addition, we also presented a method to predict the steering angle of a vehicle\nusing our model and achieve promising preliminary results on the task.\n\nAcknowledgements The work of Jiashi Feng was partially supported by National University of\nSingapore startup grant R-263-000-C08-133, Ministry of Education of Singapore AcRF Tier One\ngrant R-263-000-C21-112 and NUS IDS grant R-263-000-C67-646.\n\nReferences\n[1] Andreas Argyriou, Theodoros Evgeniou, and Massimiliano Pontil. Multi-task feature learning. In NIPS,\n\n2007.\n\n[2] Simon Baker, Daniel Scharstein, JP Lewis, Stefan Roth, Michael J Black, and Richard Szeliski. A database\nand evaluation methodology for optical \ufb02ow. International Journal of Computer Vision, 92(1):1\u201331, 2011.\n[3] Yu-Wei Chao, Jimei Yang, Brian Price, Scott Cohen, and Jia Deng. Forecasting human dynamics from\n\nstatic images. In CVPR, 2017.\n\n[4] Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L Yuille. Semantic\n\nimage segmentation with deep convolutional nets and fully connected crfs. In ICLR, 2015.\n\n[5] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson,\nUwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding.\narXiv preprint arXiv:1604.01685, 2016.\n\n[6] Theodoros Evgeniou and Massimiliano Pontil. Regularized multi\u2013task learning. In SIGKDD, 2004.\n[7] Clement Farabet, Camille Couprie, Laurent Najman, and Yann LeCun. Learning hierarchical features for\nscene labeling. Pattern Analysis and Machine Intelligence, IEEE Transactions on, 35(8):1915\u20131929, 2013.\n[8] Philipp Fischer, Alexey Dosovitskiy, Eddy Ilg, Philip H\u00e4usser, Caner Haz\u0131rba\u00b8s, Vladimir Golkov, Patrick\nvan der Smagt, Daniel Cremers, and Thomas Brox. Flownet: Learning optical \ufb02ow with convolutional\nnetworks. arXiv preprint arXiv:1504.06852, 2015.\n\n[9] David F Fouhey and C Lawrence Zitnick. Predicting object dynamics in scenes. In CVPR, 2014.\n[10] Alberto Garcia-Garcia, Sergio Orts-Escolano, Sergiu Oprea, Victor Villena-Martinez, and Jose Garcia-\nRodriguez. A review on deep learning techniques applied to semantic segmentation. arXiv preprint\narXiv:1704.06857, 2017.\n\n[11] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron\n\nCourville, and Yoshua Bengio. Generative adversarial nets. In NIPS, 2014.\n\n[12] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition.\n\nIn CVPR, 2016.\n\n9\n\n\f[13] Minh Hoai and Fernando De la Torre. Max-margin early event detectors. International Journal of Computer\n\nVision, 107(2):191\u2013202, 2014.\n\n[14] Herbert Jaeger. Tutorial on training recurrent neural networks, covering BPPT, RTRL, EKF and the\" echo\n\nstate network\" approach, volume 5. GMD-Forschungszentrum Informationstechnik, 2002.\n\n[15] Xiaojie Jin, Yunpeng Chen, Jiashi Feng, Zequn Jie, and Shuicheng Yan. Multi-path feedback recurrent\n\nneural network for scene parsing. In AAAI, 2017.\n\n[16] Xiaojie Jin, Xin Li, Huaxin Xiao, Xiaohui Shen, Zhe Lin, Jimei Yang, Yunpeng Chen, Jian Dong, Luoqi\n\nLiu, Zequn Jie, et al. Video scene parsing with predictive feature learning. In ICCV, 2017.\n\n[17] Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmenta-\n\ntion. In CVPR, 2015.\n\n[18] William Lotter, Gabriel Kreiman, and David Cox. Unsupervised learning of visual structure using predictive\n\ngenerative networks. arXiv preprint arXiv:1511.06380, 2015.\n\n[19] Zelun Luo, Boya Peng, De-An Huang, Alexandre Alahi, and Li Fei-Fei. Unsupervised learning of long-term\n\nmotion dynamics for videos. arXiv preprint arXiv:1701.01821, 2017.\n\n[20] Michael Mathieu, Camille Couprie, and Yann LeCun. Deep multi-scale video prediction beyond mean\n\nsquare error. arXiv preprint arXiv:1511.05440, 2015.\n\n[21] Rudolf Mester. Motion estimation revisited: an estimation-theoretic approach. In Image Analysis and\n\nInterpretation (SSIAI), 2014 IEEE Southwest Symposium on, pages 113\u2013116. IEEE, 2014.\n\n[22] Natalia Neverova, Pauline Luc, Camille Couprie, Jakob Verbeek, and Yann LeCun. Predicting deeper into\n\nthe future of semantic segmentation. arXiv preprint arXiv:1703.07684, 2017.\n\n[23] Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros. Context encoders:\n\nFeature learning by inpainting. In CVPR, 2016.\n\n[24] Viorica Patraucean, Ankur Handa, and Roberto Cipolla. Spatio-temporal video autoencoder with differen-\n\ntiable memory. arXiv preprint arXiv:1511.06309, 2015.\n\n[25] Jerome Revaud, Philippe Weinzaepfel, Zaid Harchaoui, and Cordelia Schmid. Epic\ufb02ow: Edge-preserving\n\ninterpolation of correspondences for optical \ufb02ow. In CVPR, 2015.\n\n[26] Anirban Roy and Sinisa Todorovic. Scene labeling using beam search under mutex constraints. In CVPR,\n\n2014.\n\n[27] Eder Santana and George Hotz. Learning a driving simulator. CoRR, abs/1608.01230, 2016.\n[28] Alexander G Schwing and Raquel Urtasun. Fully connected deep structured networks. arXiv preprint\n\narXiv:1503.02351, 2015.\n\n[29] Laura Sevilla-Lara, Deqing Sun, Varun Jampani, and Michael J Black. Optical \ufb02ow with semantic\n\nsegmentation and localized layers. In CVPR, 2016.\n\n[30] Richard Socher, Cliff C Lin, Chris Manning, and Andrew Y Ng. Parsing natural scenes and natural\n\nlanguage with recursive neural networks. In ICML, 2011.\n\n[31] Nitish Srivastava, Elman Mansimov, and Ruslan Salakhudinov. Unsupervised learning of video representa-\n\ntions using lstms. In ICML, 2015.\n\n[32] Ruben Villegas, Jimei Yang, Seunghoon Hong, Xunyu Lin, and Honglak Lee. Decomposing motion and\n\ncontent for natural video sequence prediction. In ICLR, 2017.\n\n[33] Ruben Villegas, Jimei Yang, Yuliang Zou, Sungryull Sohn, Xunyu Lin, and Honglak Lee. Learning to\n\ngenerate long-term future via hierarchical prediction. In ICML, 2017.\n\n[34] Carl Vondrick, Hamed Pirsiavash, and Antonio Torralba. Generating videos with scene dynamics. In NIPS,\n\n2016.\n\n[35] Jacob Walker, Abhinav Gupta, and Martial Hebert. Patch to the future: Unsupervised visual prediction. In\n\nCVPR, 2014.\n\n[36] Jacob Walker, Abhinav Gupta, and Martial Hebert. Dense optical \ufb02ow prediction from a static image. In\n\nICCV, 2015.\n\n[37] Jenny Yuen and Antonio Torralba. A data-driven approach for event prediction. In ECCV, 2010.\n\n10\n\n\f", "award": [], "sourceid": 3470, "authors": [{"given_name": "Xiaojie", "family_name": "Jin", "institution": "National University of Singapore & Snap Research"}, {"given_name": "Huaxin", "family_name": "Xiao", "institution": "NUDT"}, {"given_name": "Xiaohui", "family_name": "Shen", "institution": "Adobe"}, {"given_name": "Jimei", "family_name": "Yang", "institution": "Adobe Research"}, {"given_name": "Zhe", "family_name": "Lin", "institution": "Adobe"}, {"given_name": "Yunpeng", "family_name": "Chen", "institution": "National University of Singapore"}, {"given_name": "Zequn", "family_name": "Jie", "institution": "Tencent AI Lab"}, {"given_name": "Jiashi", "family_name": "Feng", "institution": "National University of Singapore"}, {"given_name": "Shuicheng", "family_name": "Yan", "institution": "Qihoo 360 AI Institute"}]}