Part of Advances in Neural Information Processing Systems 12 (NIPS 1999)
Geoffrey E. Hinton, Andrew Brown
We first show how to represent sharp posterior probability distribu(cid:173) tions using real valued coefficients on broadly-tuned basis functions. Then we show how the precise times of spikes can be used to con(cid:173) vey the real-valued coefficients on the basis functions quickly and accurately. Finally we describe a simple simulation in which spik(cid:173) ing neurons learn to model an image sequence by fitting a dynamic generative model.
1 Population codes and energy landscapes
A perceived object is represented in the brain by the activities of many neurons, but there is no general consensus on how the activities of individual neurons combine to represent the multiple properties of an object. We start by focussing on the case of a single object that has multiple instantiation parameters such as position, velocity, size and orientation. We assume that each neuron has an ideal stimulus in the space of instantiation parameters and that its activation rate or probability of activation falls off monotonically in all directions as the actual stimulus departs from this ideal. The semantic problem is to define exactly what instantiation parameters are being represented when the activities of many such neurons are specified.
Hinton, Rumelhart and McClelland (1986) consider binary neurons with receptive fields that are convex in instantiation space. They assume that when an object is present it activates all of the neurons in whose receptive fields its instantiation parameters lie. Consequently, if it is known that only one object is present, the parameter values of the object must lie within the feasible region formed by the intersection of the receptive fields of the active neurons. This will be called a con(cid:173) junctive distributed representation. Assuming that each receptive field occupies only a small fraction of the whole space, an interesting property of this type of "coarse coding" is that the bigger the receptive fields, the more accurate the repre(cid:173) sentation. However, large receptive fields lead to a loss of resolution when several objects are present simultaneously.
When the sensory input is noisy, it is impossible to infer the exact parameters of objects so it makes sense for a perceptual system to represent the probability dis(cid:173) tribution across parameters rather than just a single best estimate or a feasible region. The full probability distribution is essential for correctly combining infor-
Spiking Boltzmann Machines