Part of Advances in Neural Information Processing Systems 3 (NIPS 1990)
Jordan Pollack
A higher order recurrent neural network architecture learns to recognize and generate languages after being "trained" on categorized exemplars. Studying these networks from the perspective of dynamical systems yields two interesting discoveries: First, a longitudinal examination of the learning process illustrates a new form of mechanical inference: Induction by phase transition. A small weight adjustment causes a "bifurcation" in the limit behavior of the network. This phase transition corresponds to the onset of the network's capacity for generalizing to arbitrary-length strings. Second, a study of the automata resulting from the acquisition of previously published languages indicates that while the architecture is NOT guaranteed to find a minimal finite automata consistent with the given exemplars, which is an NP-Hard problem, the architecture does appear capable of generating non(cid:173) regular languages by exploiting fractal and chaotic dynamics. I end the paper with a hypothesis relating linguistic generative capacity to the behavioral regimes of non-linear dynamical systems.