Part of Advances in Neural Information Processing Systems 17 (NIPS 2004)
Eizaburo Doi, Michael Lewicki
It has been suggested that the primary goal of the sensory system is to represent input in such a way as to reduce the high degree of redun- dancy. Given a noisy neural representation, however, solely reducing redundancy is not desirable, since redundancy is the only clue to reduce the effects of noise. Here we propose a model that best balances redun- dancy reduction and redundant representation. Like previous models, our model accounts for the localized and oriented structure of simple cells, but it also predicts a different organization for the population. With noisy, limited-capacity units, the optimal representation becomes an overcom- plete, multi-scale representation, which, compared to previous models, is in closer agreement with physiological data. These results offer a new perspective on the expansion of the number of neurons from retina to V1 and provide a theoretical model of incorporating useful redundancy into efficient neural representations.
1 Introduction
Efficient coding theory posits that one of the primary goals of sensory coding is to eliminate redundancy from raw sensory signals, ideally representing the input by a set of statistically independent features [1]. Models for learning efficient codes, such as sparse coding [2] or ICA [3], predict the localized, oriented, and band-pass characteristics of simple cells. In this framework, units are assumed to be non-redundant and so the number of units should be identical to the dimensionality of the data.
Redundancy, however, can be beneficial if it is used to compensate for inherent noise in the system [4]. The models above assume that the system noise is low and negligible so that redundancy in the representation is not necessary. This is equivalent to assuming that the representational capacity of individual units is unlimited. Real neurons, however, have limited capacity [5], and this should place constraints on how a neural population can best encode a sensory signal. In fact, there are important characteristics of simple cells, such as the multi-scale representation, that cannot be explained by efficient coding theory.
The aim of this study is to evaluate how the optimal representation changes when the system
is constrained by limited capacity units. We propose a model that best balances redundancy reduction and redundant representation given the limited capacity units. In contrast to the efficient coding models, it is possible to have a larger number of units than the intrinsic dimensionality of the data. This further allows to introduce redundancy in the population, enabling precise reconstruction using the imprecise representation of a single unit.