{"title": "Multiple Threshold Neural Logic", "book": "Advances in Neural Information Processing Systems", "page_first": 252, "page_last": 258, "abstract": null, "full_text": "Multiple Threshold Neural Logic \n\nVasken Bohossian \n\nJehoshua Bruck \n\nCalifornia Institute of Technology \n\nMail Code 136-93 \n\nPasadena, CA 91125 \n\n~mail: {vincent, bruck}~paradise.caltech.edu \n\nAbstract \n\nWe introduce a new Boolean computing element related to the Lin(cid:173)\near Threshold element, which is the Boolean version of the neuron. \nInstead of the sign function, it computes an arbitrary (with poly(cid:173)\nnornialy many transitions) Boolean function of the weighted sum of \nits inputs. We call the new computing element an LT M element, \nwhich stands for Linear Threshold with Multiple transitions. \nThe paper consists of the following main contributions related to \nour study of LTM circuits: (i) the creation of efficient designs of \nLTM circuits for the addition of a multiple number of integers and \nthe product of two integers. In particular, we show how to compute \nthe addition of m integers with a single layer of LT M elements. \n(ii) a proof that the area of the VLSI layout is reduced from O(n2 ) \nin LT circuits to O(n) in LTM circuits, for n inputs symmetric \nBoolean functions, and (iii) the characterization of the computing \npower of LT M relative to LT circuits. \n\n1 \n\nIntroduction \n\nHuman brains are by far superior to computers in solving hard problems like combi(cid:173)\nnatorial optimization and image and speech recognition, although their basic build(cid:173)\ning blocks are several orders of magnitude slower. This observation has boosted \ninterest in the field of artificial neural networks [Hopfield 82], [Rumelhart 82]. The \nlatter are built by interconnecting artificial neurons whose behavior is inspired by \nthat of biological neurons. In this paper we consider the Boolean version of an artifi(cid:173)\ncial neuron, namely, a Linear Threshold (LT) element, which computes a neural-like \n\n\fMUltiple Threshold Neural Logic \n\n253 \n\nWI \n\nWn \n\n-Wo \n\n1 \n0 \n\nLT gate \n\nt3 \nt2 \nt1 \n\n0 \n1 \n0 \n1 \nSYM gate \n\nWI \n\nWn \n\nt3 \nt2 \nt1 \n\n0 \n1 \n0 \n1 \nLTM gate \n\nFigure 1: Schematic representation of LT, SYM and LTM computing elements. \n\nBoolean function of n binary inputs [Muroga 71]. An LT element outputs the sign \nof a weighted sum of its Boolean inputs. The main issues in the study of networks \n(circuits) consisting of LT elements, called LT circuits, include the estimation of \ntheir computational capabilities and limitations and the comparison of their prop(cid:173)\nerties with those of traditional Boolean logic circuits based on AND, OR and NOT \ngates (called AON circuits). For example, there is a strong evidence that LT cir(cid:173)\ncuits are more efficient than AON circuits in implementing a number of important \nfunctions including the addition, product and division of integers [Siu 94], [Siu 93]. \n\nMotivated by our recent work on the VLSI implementation of LT elements \n[Bohossian 95b], we introduce in this paper a more powerful computing element, \na multiple threshold neuron, which we call LTM, which stands for Linear Thresh(cid:173)\nold with Multiple transitions, see [Haring 66] and [Olafsson 88]. \nInstead of the \nsign function in the LT element it computes an arbitrary (with polynomialy many \ntransitions) Boolean function of the weighted sum of its inputs. \n\nThe main issues in the study of LTM circuits (circuits consisting of LTM elements) \ninclude the estimation of their computational capabilities and limitations and the \ncomparison of their properties to those of AON circuits. A natural approach in this \nstudy is first to understand the relation between LT circuits and LT M circuits. Our \nmain contributions in this paper are: \n\n\u2022 We demonstrate the power of LTM by deriving efficient designs of LTM \n\ncircuits for the addition of m integers and the product of two integers. \n\n\u2022 We show that LT M circuits are more amenable in implementation than LT \ncircuits. In particular, the area of the VLSI layout is reduced from O(n2 ) \nin LT circuits to O(n) in LTM circuits, for n input symmetric Boolean \nfunctions. \n\n\u2022 We characterize the computing power of LT M relative to LT circuits. \n\nNext we describe the formal definitions of LT and LT M elements. \n\n1.1 Definitions and Examples \n\nDefinition 1 (Linear Threshold Gate - LT) \nA linear threshold gate computes a Boolean function of its binary inputs : \n\nf(X) = sgn(wo + L WiXi) \n\nn \n\ni=l \n\n\f254 \n\nV. Bohossian and J. Bruck \n\nwhere the Wi are integers and sgn(.) outputs 1 if its argument is greater or equal to \n0, and 0 otherwise. \n\nFigure 1 shows an-input LT element; if L~ WiXi ~ -Wo the element outputs 1, \notherwise it outputs o. A single LT gate is unable to compute parity. The latter \nbelongs to the general class of symmetric functions - SY M. \n\nDefinition 2 (Symmetric Functions - SY M) \nA Boolean function f is symmetric if its value depends only on the number of ones \nin the input denoted by IX I. \nFigure 1 shows an example of a symmetric function; it has three transitions, it \noutputs 1 for IXI < tl and for t2 ~ IXI < t3, and 0 otherwise. AND, OR and \nparity are examples of symmetric functions. A single LT element can implement \nonly a limited subset of symmetric functions. We define LT M as a generalization \nof SY M. That is, we allow the weights to be arbitrary as in the case of LT, rather \nthan fixed to 1 (see Figure 1 ). \n\nDefinition 3 (Linear Threshold Gate with Multiple Transitions - LT M) \nA function f is in LTM if there exists a set of weights Wi E Z, 1 ~ i ~ n and a \nfunction h : Z ---+ {O, 1} such that \nn \n\nf(X) = h(L wixd for all X E {O,l}n \n\ni=l \n\nThe only constraint on h is that it undergoes polynomialy many transitions as its \ninput scans [- L~=l IWi I, L~=l IWi I]\u00b7 \nNotice that without the constraint on the number of transitions, an LTM gate is \ncapable of computing any Boolean function. Indeed, given an arbitrary function f, \nlet Wi = 2i - 1 and h(L~ 2i - 1xd = f(xt, .\u2022 \u2022 , x n ). \n\nExample 1 (XOR E LTM) \nXOR(X) outputs 1 if lXI, the number of l's in X, is odd. Otherwise it outputs \nO. To implement it choose Wi = 1 and h(k) = ~(1 - (_l)k) for 0 ~ k ~ n. Note \nthat h(k) needs not be defined for k < 0 and k > n, and has polynomialy many \ntransitions. \n\nAnother useful function that LTM can compute is ADD (X, Y), the sum of two \nn-bit integers X and Y. \n\nExample 2 (ADD E LT M) \nTo implement addition we set fl (X, Y) = h, (L~=l 2i (Xi + yd) where h, (k) = 1 for \nk E [2' ,2 x 2' - 1] U [3 X 2' , +00). Defined thus, fl computes the m-th bit of X + Y. \n\n1.2 Organization \n\nThe paper is organized as follows . In Section 2, we study a number of applications \nas well as the VLSI implementations of LTM circuits. In particular, we show how \nto compute the addition of m integers with a single layer of LT!vI elements. In \nSection 3, we prove J..he characterization results of LT M - inclusion relations, in \nparticular LTM ~ LT2. In addition, we indicate which inclusions are proper and \nexhibit functions to demonstrate the separations. \n\n\fMultiple Threshold Neural Logic \n\n2 LT M Constructions \n\n255 \n\nThe theoretical results about LT M can be applied to the VLSI implementation of \nBoolean functions. The idea of a gate with multiple thresholds came to us as we \nwere looking for an efficient VLSI implementation of symmetric Boolean functions. \nEven though a single LT gate is not powerful enough to implement any symmetric \nfunction, a 2-layer LT circuit is. FUrthermore, it is well known that such a circuit \nperforms much better than the traditional logic circuit based on AND, OR and \nNOT gates. The latter has exponential size (or unbounded depth) [Wegener 91]. \n\nProposition 4 (LT2 versus LT M for symmetric function implementation) \nThe LT2 layout of a symmetric function requires area of O(n2), while using LT M \none needs only area of O( n). \n\nPROOF: \nImplementing a generalized symmetric function in LT2 requires up to n LT gates in \nthe first layer. Those have the same weights Wi except for the threshold Woo Instead \nof laying out n times the same linear sum E~ WiXi we do it once and compare the \nresult to n different thresholds. The resulting circuit corresponds to a single LT M \ngate. \n0 \n\nThe LT2 layout is redundant, it has n copies of each weight, requiring area of at \nleast O(n2). On the other hand, LTM performs a single weighted sum, its area \nrequirement is O(n). \n\nA single LT M gate can compute the addition of m n-bit integers M ADD. The \nonly constraint is that m be polynomial in n. \n\nTheorem 5 (MADD E LTM) \nA single layer of LT M gates can compute the sum of m n-bit integers, provided \nthat m is at most polynomial in n. \n\nPROOF: \nMAD D returns an integer of at most n + log m bits. We need one LT M gate per \nbit. The least significant bit is computed by a simple m-bit XOR. For all other \nbits we use h(X(l), .. ,x(m\u00bb) = hl(E~=12i Ej=l x~j\u00bb) to compute the l-th bit ofthe \nmm. \n0 \n\nCorollary 6 (PRODUCT E PTM) A single layer of PTM (which is defined \nbelow) gates, can compute the product of m n-bit integers, provided that m is at \nmost polynomial in n. \n\nPROOF: \nBy analogy with PTb defined in [Bruck 90], in PT Ml (or simply PT M) we allow a \npolynomial rather than a linear sum: f(X) = h(WIXl + ... +wnxn +W(1,2)XIX2+ ... ) \nHowever we restrict the sum to have polynomialy many terms (else, any Boolean \nfunction could be realized with a single gate). The product of two n-bit integers \nX and Y can be written as PRODUCT(X, Y) = E~=l XiY. We use the con(cid:173)\nstruction of MADD in order to implement PRODUCT. PRODUCT(X, Y) = \nMADD(x 1Y,x2 Y , ... ,xnY). fleX, Y) = hi (LJj=l LJi=12 XjYi) b outputs the l-th \nbit of the product. \n0 \n\n\"n \"I \n\ni \n\n\f256 \n\nV. Bohossian and J Bruck \n\nFigure 2: Relationship between Classes \n\n3 Classification of LTM \n\nWe me a hat to indicate small (polynomialy growing) weights, e.g. LT, LT M \n[Bohossian 95a], [Siu 91], and a subscript to indicate the depth (number of layers) \nof the circuit of more than a single layer. All the circuits we consider in this paper \nare of polynomial size (number of elements) in n (number of inputs). For example, \nthe class fr2 consists of those B0...2!ean functions that can be implemented by a \ndepth-2 polynomial size circuit of LT elements. \n\n---\n\n.-\n\nFigure 2 depicts the membership relations between five classes of Boolean functions, \nincluding, LT, ilr, LTM, LTM and ilr2, along with the functions used to establish \nthe separations. \n\nIn this section we will prove the relations illustrated by Figure 2 . \n\nTheorem 7 (Classification of LTM ) \nThe inclusions and separations shown in Figure 12 hold. That is, \n\n. -\n\n1. LT ~ LT ; LTM \n\n12. LT ~ LTM ; LTM \n\n---\n\n.-\n\n9. LTM; LT2 \n4\u00b7 XOR E CTM but XOR tJ. LT \n5. CaMP E LT but CaMP tJ. LTM \n\n6. ADD E LTM but ADD tJ. LTULTM \n7. IPk E fr2 but 1Pk tJ. LTM \n\nPROOF: \nWe show only the outline of the proof. The complete version can be found in \n[Bohossian 96]. Claims 1 and 2 follow from the definition. The first part of Claim 4 \nwas shown in Example 1 and the second is well known. In Claim 5, CaMP stands \nfor the Comparison functio!lt the proof mes the pigeonhole principle and is related \nto the proof of CaMP tJ. LT which can be found in [Siu 91]. In Claim 6 to show \nthat ADD tJ. LTM we use the same idea as for CaMP. Claim 3 is proved using a \nresult from [Goldman 93]: a single LT gate with arbitrary weights can be realized \nby an LT2 circuit. Claim 7 introduces the function IPk(X, Y) = 1 iff L:~ XiYi ~ k, \n\n\fMultiple Threshold Neural Logic \n\n257 \n\n.-\n\no otherwise. If IPk E LTM, using the result from [Goldman 93], we can construct \na LT2 circuit that computes IP2 (Inner Product mod 2) which is known to be false \n[Hajnal 94]. \n0 \nWhat remains to be shown in order to complete the classification picture is fr = \nLT n LTM. We conjecture that this is true. \n\n4 Conclusions \n\nOur original goal was to use theoretical results in order to efficiently layout a \ngeneralized symmetric function. During that process we came to the conclusion \nthat the LT2 implementation is partially redundant, which lead to the definition \nof LTM, a new, more powerful computing element. We characterized the power \nof LTM relative to LT. We showed how it can be used to reduce the area of \nVLSI layouts from O(n2 ) to O(n) and derive efficient designs for multiple addition \nand product. Interesting directions for future investigation are (i) to prove the \nconjecture: fr = LT n LTM, (ii) to apply spectral techniques ([Bruck 90)) to the \nanalysis of LT M, in particular show how PT M fits into the classification picture \n(Figure 2 ). \n\nAnother direction for future research consists in introducing the ideas described \nabove in the domain of VLSI. We have fabricated a programmable generalized \nsymmetric function on a 2J,L, analog chip using the model described above. Floating \ngate technology is used to program the weights. We store a weight on a single \ntransistor by injecting and tunneling electrons on the floating gate [Hasler 95]. \n\nAcknowledgments \n\nThis work was supported in part by the NSF Young Investigator Award CCR-\n9457811 and by the Sloan Research Fellowship. \n\nReferences \n\n[Bohossian 95a] V. Bohossian and J. Bruck. On Neural Networks with Minimal \nWeights. In Advances in Neural Information Processing Systems 8, MIT Press, \nCambridge, MA, 1996, pp.246-252. \n\n[Bohossian 95b] V. Bohossian, P. Hasler and J. Bruck. Programmable Neural Logic. \nProceedings of the second annual IEEE International Conference on Innovative \nSystems in Silicon, pp. 13-21, October 1997. \n\n[Bohossian 96] V. Bohossian and J. Bruck. \nTechnical Report, ETR010, \n\nral Logic. \nhttp://paradise.caltech.edu/ETR.html) \n\nMultiple Threshold N eu-\n(available at \n\nJune 1996. \n\n[Bruck 90] J. Bruck. Harmonic Analysis of Polynomial Threshold Functions. SIAM \n\nJ. Disc. Math, Vol. 3(No. 2)pp. 168- 177, May 1990. \n\n[Goldman 93] M. Goldmann and M. Karpinski. Simulating threshold circuits by \n\nmajority circuits. In Proc. 25th ACM STOC, pages pp. 551-560, 1993. \n\n\f258 \n\nV. Bohossian and J Bruck \n\n[Hajnal 94] A. Hajnal, W. Maass, P. Pudlak, M. Szegedy, G. Turan. Threshold \nCircuits of Bounded Depth. Journal of Computer and System Sciences, Vol. \n46(No. 2):pp. 129-154, April 1993. \n\n[Haring 66] D.R. Haring. Multi-Threshold Threshold Elements. IEEE Transactions \n\non Electronic Computers, Vol. EC-15, No.1, February 1966. \n\n[Hasler 95] P. Hasler, C. Diorio, B.A. Minch and C.A. Mead. Single Transistor \nLearning Synapses. Advances in Neural Information Processing Systems 7, \nMIT Press, Cambridge, MA, 1995, pp.817-824. \n\n[Hastad 94] J. Hastad. On the size of weights for threshold gates. SIAM. J. Disc. \n\nMath., 7:484-492, 1994. \n\n[Hofmeister 96] T. Hofmeister. A Note on the Simulation of Exponential Threshold \n\nWeights. 1996 CONCOON conference. \n\n[Hopfield 82] J. Hopfield. Neural networks and physical systems with emergent col(cid:173)\n\nlective computational abilities. Proc. of the USA National Academy of Sciences, \n79:2554-2558, 1982. \n\n[Muroga 71] M. Muroga. Threshold Logic and its Applications. Wiley-Interscience, \n\n1971. \n\n[Olafsson 88] S. Olafsson and Y.S. Abu-Mostafa. The Capacity of Multilevel \nThreshold Functions. IEEE Transactions on Pattern Analysis and Machine \nIntelligence, Vol.10, No.2, March 1988. \n\n[Rumelhart 82] D. Rumelhart and J. McClelland. Parallel distributed processing: \n\nExplorations in the microstructure of cognition. MIT Press, 1982. \n\n[Siu 91] K. Siu and J. Bruck. On the power of threshold circuits with small weights. \n\nSIAM J. Disc. Math., Vol. 4(No. 3):pp. 423-435, August 1991. \n\n[Siu 93] K. Siu, J. Bruck, T. Kailath, and T. Hofmeister. Depth Efficient Neural \nNetworks for Division and Related Problems. IEEE Trans. on Information \nTheory, Vol. 39(No. 3), May 1993. \n\n[Siu 94] K. Siu and V.P. Roychowdhury. On Optimal Depth Threshold Circuits for \nMultiplication and Related Problems. SIAM J. Disc. Math., Vol. 7(No. 2):pp. \n284-292, May 94. \n\n[Szegedy 89] M. Szegedy. Algebraic Methods in Lower Bounds for Computational \nModels with Limited Communication. PhD Thesis, Dep. Computer Science, \nChicago Univ., December 1989. \n\n[Wegener 91] 1. Wegener. The complexity of the parity function in unbounded fan(cid:173)\n\nin unbounded depth circuits. In Theoretical Computer Science, Vol. 85, pp. \n155-170, 1991. \n\n\f", "award": [], "sourceid": 1396, "authors": [{"given_name": "Vasken", "family_name": "Bohossian", "institution": null}, {"given_name": "Jehoshua", "family_name": "Bruck", "institution": null}]}