{"title": "Fully Neural Network based Model for General Temporal Point Processes", "book": "Advances in Neural Information Processing Systems", "page_first": 2122, "page_last": 2132, "abstract": "A temporal point process is a mathematical model for a time series of discrete events, which covers various applications. Recently, recurrent neural network (RNN) based models have been developed for point processes and have been found effective. RNN based models usually assume a specific functional form for the time course of the intensity function of a point process (e.g., exponentially decreasing or increasing with the time since the most recent event). However, such an assumption can restrict the expressive power of the model. We herein propose a novel RNN based model in which the time course of the intensity function is represented in a general manner. In our approach, we first model the integral of the intensity function using a feedforward neural network and then obtain the intensity function as its derivative. This approach enables us to both obtain a flexible model of the intensity function and exactly evaluate the log-likelihood function, which contains the integral of the intensity function, without any numerical approximations. Our model achieves competitive or superior performances compared to the previous state-of-the-art methods for both synthetic and real datasets.", "full_text": "Fully Neural Network based Model for General\n\nTemporal Point Processes\n\nTakahiro Omi\n\nThe University of Tokyo, RIKEN AIP\n\ntakahiro.omi.em@gmail.com\n\nNaonori Ueda\n\nNTT Communication Science Laboratories, RIKEN AIP\n\nnaonori.ueda.fr@hco.ntt.co.jp\n\nKazuyuki Aihara\n\nThe University of Tokyo\n\naihara@sat.t.u-tokyo.ac.jp\n\nAbstract\n\nA temporal point process is a mathematical model for a time series of discrete\nevents, which covers various applications. Recently, recurrent neural network\n(RNN) based models have been developed for point processes and have been\nfound effective. RNN based models usually assume a speci\ufb01c functional form\nfor the time course of the intensity function of a point process (e.g., exponentially\ndecreasing or increasing with the time since the most recent event). However, such\nan assumption can restrict the expressive power of the model. We herein propose a\nnovel RNN based model in which the time course of the intensity function is rep-\nresented in a general manner. In our approach, we \ufb01rst model the integral of the\nintensity function using a feedforward neural network and then obtain the intensity\nfunction as its derivative. This approach enables us to both obtain a \ufb02exible model\nof the intensity function and exactly evaluate the log-likelihood function, which\ncontains the integral of the intensity function, without any numerical approxima-\ntions. Our model achieves competitive or superior performances compared to the\nprevious state-of-the-art methods for both synthetic and real datasets.\n\n1\n\nIntroduction\n\nThe activity of many diverse systems is characterized as a sequence of temporally discrete events.\nThe examples include \ufb01nancial transactions, communication in a social network, and user activity\nat a web site. In many cases, the occurrences of the event are correlated to each other in a certain\nmanner, and information on future events may be extracted from the information of past events.\nTherefore, the appropriate modeling of the dependence of the event occurrence on the history of\npast events is important for understanding the system and predicting future events.\nA temporal point process is a useful mathematical tool for modeling the time series of discrete\nevents. In this framework, the dependence on the event history is characterized using a conditional\nintensity function that maps the history of the past events to the intensity function of the point\nprocess. The most common models, such as the Poisson process or the Hawkes process [1, 2, 3],\nassume a speci\ufb01c parametric form for the conditional intensity function. Recently, Du et al. (2016)\nproposed a model based on a recurrent neural network (RNN) for point processes [4], and the variant\nmodels were further developed [5, 6, 7, 8, 9, 10]. In this approach, an RNN is used to obtain a\ncompact representation of the event history. The conditional intensity function is then modeled as\na function of the hidden state of the RNN. Consequently, the RNN based models outperform the\nparametric models in prediction performance.\n\n33rd Conference on Neural Information Processing Systems (NeurIPS 2019), Vancouver, Canada.\n\n\fAlthough such RNN based models aim to capture the dependence of the event occurrence on the\nevent history in a general manner, a speci\ufb01c functional form is usually assumed for the time course of\nthe conditional intensity function (see [6, 7] for exception). For example, the model in [4] assumed\nthat the conditional intensity function exponentially decreases or increases with the elapsed time\nfrom the most recent event until the next event. However, using such an assumption can limit\nthe expressive ability of the model and potentially deteriorate the predictive skill if the employed\nassumption is incorrect. We herein generalize RNN based models such that the time evolution of the\nconditional intensity function is represented in a general manner. For this purpose, we formulate the\nconditional intensity function based on a neural network rather than assuming a speci\ufb01c functional\nform.\nHowever, exactly evaluating the log-likelihood function for such a general model is generally in-\ntractable because the log-likelihood function of a temporal point process contains the integral of\nthe conditional intensity function. Although some studies, which considered a general model of\nthe intensity function, used numerical approximations to evaluate the integral [6, 7], numerical ap-\nproximations can deteriorate the \ufb01tting accuracy and can also be computationally expensive. To\novercome this limitation, we \ufb01rst model the integral of the conditional intensity function using a\nfeedforward neural network rather than directly modeling the conditional intensity function itself.\nThen, the conditional intensity function is obtained by differentiating it. This approach enables us to\nexactly evaluate the log-likelihood function of our general model without numerical approximations.\nFinally, we show the effectiveness of our proposed model by analyzing synthetic and real datasets.\n\n2 Method\n\n2.1 Temporal point process\n\nA temporal point process is a stochastic process that generates a sequence of discrete events at times\nftign\ni=1 in a given observation interval [0; T ]. The process is characterized via a conditional intensity\nfunction (cid:21)(tjHt), which is the intensity function of the event at the time t conditioned on the event\nhistory Ht = ftijti < tg up to the time t, given as follows:\n\n(cid:21)(tjHt) = lim\n\u2206!0\n\nP (one event occurs in [t; t + \u2206)jHt)\n\n\u2206\n\n:\n\n(1)\n\n\u222b\n\n{\n\nIf the conditional intensity function is speci\ufb01ed, the probability density function of the time ti+1 of\nthe next event, given the times ft1; t2; : : : ; tig of the past events, is obtained as follows:\n\n}\n(cid:21)(tjHt)dt\n}\nwhere the exponential term in the right-hand side represents the probability that no events occur in\n[ti; ti+1). The probability density function to observe an event sequence ftign\ni=1 is then obtained as\nfollows:\n(cid:21)(tjHt)dt\n\np(ti+1jt1; t2; : : : ; ti) = (cid:21)(ti+1jHti+1) exp\n{\n\n(cid:21)(tijHti) exp\n\np(ftign\n\ni=1) =\n\n\u222b\n\nti+1\n\n(cid:0)\n\nti\n\n;\n\n(2)\n\n:\n\n(3)\n\n(cid:0)\n\nT\n\n0\n\nn\u220f\n\ni=1\n\nThe most basic example of a temporal point process is a stationary Poisson process, which assumes\n\u2211\nthat the events are independent of each other. The conditional intensity function of the stationary\nPoisson process is given as (cid:21)(tjHt) = (cid:21). Another popular example is the Hawkes process [1, 2, 3],\nwhich is a simple model of a self-exciting point process. The conditional intensity function of the\nti