Part of Advances in Neural Information Processing Systems 7 (NIPS 1994)
Kam Jim, Bill Horne, C. Giles
We introduce and study methods of inserting synaptic noise into dynamically-driven recurrent neural networks and show that ap(cid:173) plying a controlled amount of noise during training may improve convergence and generalization. In addition, we analyze the effects of each noise parameter (additive vs. multiplicative, cumulative vs. non-cumulative, per time step vs. per string) and predict that best overall performance can be achieved by injecting additive noise at each time step. Extensive simulations on learning the dual parity grammar from temporal strings substantiate these predictions.