{"title": "Recurrent Networks and NARMA Modeling", "book": "Advances in Neural Information Processing Systems", "page_first": 301, "page_last": 308, "abstract": null, "full_text": "Recurrent Networks and N ARMA Modeling \n\nJerome Connor \n\nLes E. Atlas \n\nDouglas R. Martin \n\nFT-lO \n\nInteractive Systems Design Laboratory \n\nDept. of Electrical Engineering \n\nUniversity of Washington \nSeattle, Washington 98195 \n\nB-317 \n\nDept. of Statistics \n\nUniversity of Washington \nSeattle, Washington 98195 \n\nAbstract \n\nThere exist large classes of time series, such as those with nonlinear moving \naverage components, that are not well modeled by feedforward networks \nor linear models, but can be modeled by recurrent networks. We show that \nrecurrent neural networks are a type of nonlinear autoregressive-moving \naverage (N ARMA) model. Practical ability will be shown in the results of \na competition sponsored by the Puget Sound Power and Light Company, \nwhere the recurrent networks gave the best performance on electric load \nforecasting. \n\n1 \n\nIntroduction \n\nThis paper will concentrate on identifying types of time series for which a recurrent \nnetwork provides a significantly better model, and corresponding prediction, than \na feedforward network. Our main interest is in discrete time series that are par(cid:173)\nsimoniously modeled by a simple recurrent network, but for which, a feedforward \nneural network is highly non-parsimonious by virtue of requiring an infinite amount \nof past observations as input to achieve the same accuracy in prediction. \nOur approach is to consider predictive neural networks as stochastic models. Section \n2 will be devoted to a brief summary of time series theory that will be used to \nillustrate the the differences between feedforward and recurrent networks. Section 3 \nwill investigate some of the problems associated with nonlinear moving average and \nstate space models of time series. In particular, neural networks will be analyzed as \n301 \n\n\f302 \n\nConnor, Atlas, and Martin \n\nnonlinear extensions oftraditionallinear models. From the preceding sections, it will \nbecome apparent that the recurrent network will have advantages over feedforward \nneural networks in much the same way that ARMA models have over autoregressive \nmodels for some types of time series. \nFinally in section 4, the results of a competition in electric load forecasting spon(cid:173)\nsored by the Puget Sound Power and Light Company will discussed. In this com(cid:173)\npetition, a recurrent network model gave superior results to feed forward networks \nand various types of linear models. The advantages of a state space model for \nmultivariate time series will be shown on the Puget Power time series. \n\n2 Traditional Approaches to Time Series Analysis \n\nThe statistical approach to forecasting involves the construction of stochastic mod(cid:173)\nels to predict the value of an observation Xt using previous observations. This is \noften accomplished using linear stochastic difference equation models, with random \ninputs. \nA very general class of linear models used for forecasting purposes is the class of \nARMA(p,q) models \n\np \n\nq \n\nXt = L