{"title": "Explorations with the Dynamic Wave Model", "book": "Advances in Neural Information Processing Systems", "page_first": 549, "page_last": 555, "abstract": null, "full_text": "Explorations with the Dynamic Wave \n\nModel \n\nThomas P. Rebotier \n\nDepartment of Cognitive Science \n\nUCSD, 9500 Gilman Dr \n\nLA JOLLA CA 92093-0515 \n\nrebotier@cogsci.ucsd.edu \n\nJeffrey L. Elman \n\nDepartment of Cognitive Science \n\nUCSD, 9500 Gilman Dr \n\nLA JOLLA CA 92093-0515 \n\nelman@cogsci.ucsd.edu \n\nAbstract \n\nFollowing Shrager and Johnson (1995) we study growth of logi(cid:173)\ncal function complexity in a network swept by two overlapping \nwaves: one of pruning, and the other of Hebbian reinforcement of \nconnections. Results indicate a significant spatial gradient in the \nappearance of both linearly separable and non linearly separable \nfunctions of the two inputs of the network; the n.l.s. cells are much \nsparser and their slope of appearance is sensitive to parameters in \na highly non-linear way. \n\n1 \n\nINTRODUCTION \n\nBoth the complexity of the brain (and concomittant difficulty encoding that C0111-\nplexity through any direct genetic mapping). as well as the apparently high degree \nof cortical plasticity suggest that a great deal of cortical structure is emergent \nrather than pre-specified. Several neural models have explored the emergence of \ncomplexity. Von der Marlsburg (1973) studied the grouping of orientation selec(cid:173)\ntivity by competitive Hebbian synaptic modification. Linsker (1986.a, 1986 .b and \n1986.c) showed how spatial selection cells (off-center on-surround), orientation selec(cid:173)\ntive cells, and finally orientation columns, emerge in successive layers from random \ninput by simple, Hebbian-like learning rules . ~[iller (1992, 1994) studied the emer(cid:173)\ngence of orientation selective columns from activity dependant competition between \non-center and off-center inputs. \n\nKerzsberg, Changeux and Dehaene (1992) studied a model with a dual-aspect learn(cid:173)\ning mechanism: Hebbian reinforcement of the connection strengths in case of cor(cid:173)\nrelated activity, and gradual pruning of immature connections. Cells in this model \nwere organized on a 2D grid , connected to each other according to a probability ex(cid:173)\nponentially decreasing with distance , and received inputs from two different sources, \n\n\f550 \n\nT. P. REBOTIER, J. L. ELMAN \n\nA and B, which might or might not be correlated. The analysis of the network re(cid:173)\nvealed 17 different kinds of cells: those whose ou tpu t after several cycles depended \non the network's initial state, and the 16 possible logical functions of two inputs. \nKerzsberg et al. found that learning and pruning created different patches of cells \nimplementing common logical functions, with strong excitation within the patches \nand inhibition between patches. \n\nShrager and Johnson (1995) extended that work by giving the network structure in \nspace (structuring the inputs in intricated stripes) or in time, by having a Hebbian \nlearning occur in a spatiotemporal wave that passed through the network rather \nthan occurring everywhere simultaneously. Their motivation was to see if these \nlearning conditions might create a cascade of increasingly complex functions. The \napproach was also motivated by developmental findings in humans and monkeys \nsuggesting a move of the peak of maximal plasticity from the primary sensory \nand motor areas to\\vards parietal and then frontal regions. Shrager and Johnson \nclassified the logical functions into three groups: the constants (order 0), those that \ndepend on one input only (order 1), those that depend on both inputs (order 2). \nThey found that a slow wave favored the growth of order 2 cells, whereas a fast \nwave favored order 1 cells. However, they only varied the connection reinforcement \n(the growth Trophic Factor), so that the still diffuse pruning affected the rightmost \nconnections before they could stabilize, resulting in an overall decrease which had \nto be compensated for in the analysis. \n\nIn this work, v,,'e followed Shrager and Johnson in their study of the effect of a \ndynamic wave of learning. We present three novel features. Firstly, both the growth \ntrophic factor (hereafter, TF) and the probability of pruning (by analogy, \"death \nfactor\", DF) travel in gaussian-shaped waves. Second, we classify the cells in 4, not \n3, orders: order 3 is made of the non-linearly separable logical functions, whereas \nthe order 2 is now restricted to linearly separable logical functions of both inputs. \nThird. we use an overall measure of network performance: the slope of appearance \nof units of a given order. The density is neglected as a measure not related to the \nspecific effects we are looking for, namely, spatial changes in complexity. Thus, each \nrun of our network can be analyzed using 4 values: the slopes for units of order 0, \n1, 2 and 3 (See Table 1.). This extreme summarization of functional information \nallows us to explore systematically many parameters and to study their influence \nover how complexity grows in space. \n\nTable 1: Orders of logical complexity \n\nORDER \no \n1 \n2 \n3 \n\nFUNCTIONS \n\nTrue False \nA !A B !B \nA.B !A.B A.!B !A.!B AvB !AvB Av!B !Av!B \nA xor B, A==B \n\n2 METHODS \n\nOur basic network consisted of 4 columns of 50 units (one simulation verified the \nscaling up of results, see section 3.2). Internal connections had a gaussian band(cid:173)\nwidth and did not wrap around . All initial connections were of weight 1, so that the \nconnectivity weights given as parameters specified a number of labile connections. \nEarly investigations were made with a set of manually chosen parameters (\" MAN-\n\n\fExplorations with the Dynamic Wave Model \n\n551 \n\nUAL\"). Afterwards, two sets of parameters were determined by a Genetic Algorithm \n(see Goldberg 1989): the first, \"SYM\", by maximizing the slope of appearance of \norder 3 units only, the second, \" ASY\" , byoptimizing jointly the appearance of order \n2 and order 3 units (\" ASY\"). The \"SYM\" network keeps a symmetrical rate of \npresentation between inputs A and B. In contrast, the\" ASY\" net presents input \nB much more often than input A. Parameters are specified in Table 1 and, are in \n\"natural\" units: bandwidths and distances are in \"cells apart\", trophic factor is \nhomogenous to a weight, pruning is a total probability. Initial values and prun(cid:173)\ning necessited random number generation. \\Ve used a linear congruence generator \n(see p284 in Press 1988), so that given the same seed, two different machines could \nproduce exactly the same run. All the points of each Figure are means of several \n(usually 40) runs with different random seeds and share the same series of random \nseeds. \n\nTable 2: Default parameters \n\nMAN. SYM. ASY. name \n\ndescription \n\n8.5 \n6.5 \n8.5 \n6.5 \n5.0 \n3.5 \n0.2 \n7.0 \n7.0 \n0.7 \n1.5 \n9.87 \n0.6 \n3.5 \n0.6 \n0.65 \n0.5 \n0.5 \n0.00 \n\n6.20 \n5.2 \n8.5 \n6.5 \n6.5 \n1.24 \n0.20 \n1.26 \n2.86 \n0.68 \n3.0 \n17.6 \n0.6 \n1.87 \n0.64 \n0.62 \n0.5 \n0.5 \n0.00 \n\nWae mean ini. weight of A excitatory connections \n12 \nWai mean ini. weight of A inhibitory connections \n9.7 \n13.4 Wbe mean ini. weight of B excitatory connections \n\\Vbi mean ini. weight of B inhibitory connections \n14.1 \nWne m.ini. density of internal excitatory connections \n9.9 \n12.4 Wni m.ini. density of internal inhibitory connections \nDW \n0.28 \nBne \n0.65 \nBni \n0.03 \nCdw \n0.98 \n-3.2 \nDdw \n16.4 Wtf \nBtf \n0.6 \nTst \n3.3 \nBdf \n0.5 \n0.12 \nPdf \nPa \n0.06 \nPb \n0.81 \n0.00 \nPab \n\nrelative variation in initial weights \nbandwidth of internal excitatory connections \nbandwidth of internal inhibitory connections \ncelerity of dynamic wave \ndistance between the peaks of both waves \nbase level of TF (=highest available weight) \nbandwidth of TF dynamic wave \nThreshold of stabilisation (pruning stop) \nband .. vidth of DF dynamic wave \nbase level of DF (total proba. of degeneration) \nprobability of A alone in the stimulus set \nprobability of B alone in the stimulus set \nprobability of simultaneous s A and B \n\n3 RESULTS \n\n3.1 RESULTS FORMAT \n\nAll Figures have the same format and summarize 40 runs per point unless otherwise \nspecified. The top graph presents the mean slope of appearance of all 4 orders \nof complexity (see Table 1) on the y axis , as a function of different values of the \nexperimentally manipulated parameter, on the x axis. The bottom left graph shows \nthe mean slope for order 2, surrounded by a gray area one standard deviation below \nand above. The bottom right graph shows the mean slope for order 3, also with \na I-s.d. surrounding area. The slopes have not been normalized, and come from \nnetworks whose columns are 50 units high, so that a slope of 1.0 indicates that the \nnumber of such units increase in average by one unit per columns, ie, by 3 units \n\n\f", "award": [], "sourceid": 1142, "authors": [{"given_name": "Thomas", "family_name": "Rebotier", "institution": null}, {"given_name": "Jeffrey", "family_name": "Elman", "institution": null}]}