Part of Advances in Neural Information Processing Systems 1 (NIPS 1988)
Stephen Hanson, Lorien Pratt
learning
representations during
Rumelhart (1987). has proposed a method for choosing minimal or "simple" in Back-propagation networks. This approach can be used to (a) dynamically select the number of hidden units. (b) construct a representation that is appropriate for the problem and (c) thus improve the generalization ability of Back-propagation networks. The method Rumelhart suggests involves adding penalty terms to the usual error function. In this paper we introduce Rumelhart·s minimal networks idea and compare two possible biases on the weight search space. These biases are compared in both simple counting problems and a speech recognition problem. In general. the constrained search does seem to minimize the number of hidden units required with an expected increase in local minima.