Part of Advances in Neural Information Processing Systems 5 (NIPS 1992)
Janet Wiles, Mark Ollila
Hidden units in multi-layer networks form a representation space in which each region can be identified with a class of equivalent outputs (Elman, 1989) or a logical state in a finite state machine (Cleeremans, Servan-Schreiber & McClelland, 1989; Giles, Sun, Chen, Lee, & Chen, 1990). We extend the analysis of the spatial structure of hidden unit space to a combinatorial task, based on binding features together in a visual scene. The logical structure requires a combinatorial number of states to represent all valid scenes. On analysing our networks, we find that the high dimensionality of hidden unit space is exploited by using the intersection of neighboring regions to represent conjunctions of features. These results show how combinatorial structure can be based on the spatial nature of networks, and not just on their emulation of logical structure.
1 TECHNIQUES FOR ANALYSING THE SPATIAL AND LOGICAL STRUCTURE OF HIDDEN UNIT SPACE
In multi-layer networks, regions of hidden unit space can be identified with classes of equivalent outputs. For example, Elman (1989) showed that the hidden unit patterns for words in simple grammatical sentences cluster into regions, with similar patterns representing similar grammatical entities. For example, different tokens of the same word are clustered tightly, indicating that they are represented within a small region. These regions can be grouped into larger regions, reflecting a hierarchical structure. The largest