Equation 1
Progress Report For Contract
ONR- N00014-97-C-0049
"Biological Extensions to NeuroSolutions"
for the period
March 1, 1997 - April 30, 1997
Prepared by
Curt Lefebvre
NeuroDimension, Incorporated
1800 North Main Street, D4
Gainesville, FL 32609
May 2, 1997
1. Introduction
This report is divided into three parts: The continuation of the implementation of Freeman’s dynamic processing element in NeuroSolutions, the design of the modifications to be incorporated in NeuroSolutions and finally some organizational issues.
2. Digital Implementation of Freeman’s dynamic PE
In our previous report, we established a new method to digitally simulate Freeman’s PE which avoids 4th order Runge-Kutta integration. Runge-Kutta integration is almost exclusively utilized for the simulation of continuous time dynamic models. Although the method is quite general there is a tremendous computational cost associated with the method, which hinders its applicability. Either small models or very expensive computers are required to implement the Runge-Kutta integration in a reasonable time.
Our method is based on digital signal processing (DSP) principles, which have been successfully applied to similar problems in speech recognition. In the early 60’s the need developed to simulate the vocal track to produce speech. The first attempts were exactly based on numeric integration of the continuous time models, but soon researchers realized that they were too time consuming for practical use. The idea developed was to discretize the model instead of discretizing time as done in numeric integration. This gave rise to the expanding field of DSP. For reasons that can be traced to the difficulty of biological models, as well as to the weak interaction between the fields of DSP and biology, no one has applied DSP principles to simulate biological models. The method we proposed in the last report applies to nonlinear models of a special type, where the dynamics are defined by linear dynamic models with real poles and the nonlinearity is static. Although this is a subset of all possible dynamical models, there are a lot of practical cases in biology that can be modeled by these assumptions, as Freeman’s KII model [1], which we called the Freeman PE. The technique basically formulates an eigendecomposition of the linear dynamical system and creates a recursive digital implementation of the continuous time linear system. We showed that the technique matches the impulse response of the dynamical system, and that it was implemented in NeuroSolutions as proposed. The question that was raised in the last report was the following: when we connect several Freeman PEs does the digital simualtion behave as the Runge-Kutta simulation. Notice that due to the fact that the connection of Freeman PEs yield a chaotic system, minor modifications can in fact affect performance. We will report next the tests conducted to answer this question.
2.1 Comparison with Runge-Kutta simulation of the KIII model
For the present discussion we are concerned with a digital implementation of the full KIII Network, proposed by Freeman [2] as a model for the olfactory bulb. The system in discussion is composed of simple non-linear blocks with second order (linear) dynamics. Freeman’s model is highly appealing from an engineering point of view since it uses chaotic attractors to encode information. This is new to the field of Artificial Neural Networks (ANN) and should be explored in terms of signal processing. We believe that this will emerge as a new class of ANNs with more powerful recognition capabilities for time varying phenomena, exactly the type of problems that conventional ANNs do not perform well.
The KIII model is subdivided into subnets namely K0, KI, KII (which we called the Freeman PE). Below we present results for the KII cell which is an eighth order oscillator created from 4 K0 cells (Figure 1) which dynamics are described by a linear part given by Eq.1
Equation 1
where a and b are constants, and x(t) represents the membrane voltage, followed by a nonlinear part y=Q(x) where
Equation 2
We chose the d operator as the transformation from the continuous time domain to the digital domain, i.e. we substituted the derivative by the operator
Equation 3
This choice is based on the argument that we want to preserve the dynamics as much as possible [5]. From this equation, and if we take a limit when TÕ 0, the d operator degenerate into the continuos time differential operator. This means that with a suitable choice of T we may have a very good approximation of the continuous time dynamics. In DSP oversampling by ten (i.e. the sampling frequency is ten times higher than the Nyquist rate) is normally considered a good approximation. However at this point we do not know if it is sufficient for the KIII since it has a chaotic behavior, but, as the results will show, this is a fairly good choice for the KII cell (Figure 1).
.
Figure 1. Freeman’s PE (KII Model).
In order to compare the accuracy of this modification we simulated the same Freeman PE with a 4th order Runge-Kutta adaptive stepsize integration. The MATLAB function was directly utilized. The Freeman PE was driven with 1/0/1 input and the results plotted (Figure 2).
The top panel shows the system response (the excitactory PE) for 3 seconds along with the phase space plot (t=1). The slope of the curve basically gives us the frequency of oscillation. In the bottom panel we also show the response of the same system utilizing the delta operator.
Figure 2. Comparison of Runge-Kutta with an operator model.
As we can see the two plots are very similar. We can observe that the decay of the delta operator response is slightly different from the one obtained with Runge-Kutta, but the overall shape of the response is the same. We also can see that the phase plot is slanted as exactly the same angle showing that the delays have been preserved.
Our next experiment was to create an interconnection of two Freeman PEs using Hebbian learning to show that the two systems can be already simulated in NeuroSolutions. This is shown in the following Figure 3.
Figure 3. NeuroSolutions breadboard with 2 Freeman PEs.
The major concern is that these are just 2 Freeman PEs and the screen is already fully populated with components what makes it very hard to read, interpret and configure reasonable size networks. This breadboard was trained using normalized Hebbian learning (Oja’s rule) and the result is shown in the figure below
Figure 4. Output of simulations with the two PEs coupled.
This result was obtained with the two Freeman PEs coupled as shown in the previous figure and after 2,000 iterations of Hebbian learning. The input was two steady state values mimicking activation of both inputs. Each trace represents one of the four outputs of Freeman’s model (first 4 correspond to the first PE). As we can see the two systems are coupled and display a totally different behavior than when they are separated. One of the objectives of the remaining of the project is to find out if the patterns that we are obtaining make sense by producing in MATLAB a Runge-Kutta simulation of the same system with the same parameters. We acknowledge that our HebbianSynapse fully couples the outputs which is not the way Freeman trains his system [2]. A new component has to be developed to exactly implement the learning.
This example shows that we have achieved one of the objectives of this Phase I proposal, i.e. the simulation in NeuroSolutions of a biological plausible neural network.
3. Modifications to the NeuroSolutions Package
During this period we also designed further modifications that need to be included in NeuroSolutions to implement Freeman’s model. Three of the most fundamental, which are the collapsing of breadboards, the asynchronous firing and the selective Hebbian synapse will be described next.
3.1 Collapsing of breadboards
One of the problems faced by this research is the differences between Freeman’s PE and the conventional PEs used in artificial neural networks. Here the PE is a dynamical system with second order dynamics while in ANNs the PE is static (as in the multilayer perceptron) or at most first order recurrent. So implementing Freeman’s PE in NeuroSolutions required an assessment of the facilities available. We show in Figure 5 the implementation of Freeman’s PE with NeuroSolutions components.
Figure 5. Freeman PE in NeuroSolutions.
It requires a GammaMemory, an ArbitrarySynapse (specifically designed for this project) a NonlinearAxon (nonlinearity with biological plausibility which was not implemented yet) and a FullSynapse. We would be severely limited by the screen size and construction time if we had to built a large system "by hand". This is goal of the enhancement to the package which we called collapsing breadboards. The idea is the following:
Construct a breadboard by hand in NeuroSolutions (as Figure 5) and then collapse it to a single component. This single component is a super-icon which will contain all the functionality of the existing breadboard, and most importantly can be replicated and recognized by the rest of the package as a new object that inherit the properties of the old ensemble and preserves the interconnection. First of all, this is possible due to the object-oriented design of NeuroSolutions.
We had to specify this module and the user dialog that enables the specification of the inputs, outputs, and probe points. We had to make several decisions for project feasibility in the time allocated. For instance, at this point we decided not to include learning in the enhancement. This decision will have to be lifted for future work, but for our application will not affect the testing of the idea since the Freeman PE has NO INTERNAL ADAPTABLE parameters. The only adaptable parameters in the full network are the connections among the excitactory and inhibitory neurons of the Freeman PEs, using Hebbian learning. This can still be done using a FullSynapse as Figure 3 shows. Presently the module is being written in C++ and is under test.
3.2. Asynchronous Firing
This is the other important modification in the package that we have designed. The problem is related to the digital simulation of recurrent systems, and it is not new, although it is insufficiently studied. We can explain it in the following way. Recurrent systems in continuous time are truly parallel. Let us take Figure 3 as an example. The influence of the output of PE 1 in PE2 happens at the same time as the reverse influence. This can not be EVER implemented digitally since in discrete time there is a delay of one sample between the output of one system and the return of its effect, otherwise the digital system will not be causal. So how can we solve this problem? NeuroSolutions as most simulators that are written for efficiency uses a synchronous update which can be implemented by matrix operations. What this means is that PE 1 fires when all its inputs have been received, which implies that there is a synchronization step produced by the firings. In Hopfield nets researchers complained that the behavior of the digital networks were at times different from the continuous time systems, in particular after switching. Synchronization affects the short term dynamics of the system, which can cause problems when the energy landscapes are complex and/or the systems have complex dynamics (such as chaotic dynamics).
An alternative that was proposed is the asynchronous update. In asynchronous update, each input to the PE is propagated forward and the next firing already includes the effect of the first update. If this is done sequentially no real advantage is noticed, but if the firing of the inputs is done randomly a more complex behavior similar to the continuous time is obtained. Our goal is to include asynchronous update in NeuroSolutions and test the difference in performance between the two update regimes.
In order to modify the program we have to work at the dataflow level, which is deep inside the core of the package. Since we do not know before hand the effect of this modification on perfromance, we would like to first simulate the modification and judge its effectiveness. Therefore we decided to develop a DLL (Dynamic Link Library) that will mimic the effect of the random firing of components. This solution is interim, since it is very inefficient. Basically what the DLL will do is to fire all the inputs, but randomly set to zero all inputs except one. This accomplishes the basic function of asynchronous update, but does it at the cost of increasing the simulation time proportionally to the number of inputs to every component. Potentially this is impractical for large networks, but we would like to judge the necessity of this deep modification in the core of the package before we actually do it. This DLL is presently being implemented and tested.
Selective Hebbian learning
In Freeman’s model, the connections between KII models are the only ones adapted. All the rest of the interconnected matrix is fixed by anatomical considerations. The NeuroSolutions components that perform Hebbian learning are the HebbSynapse and the OjaSynapse. The advantage of Oja’s rule is that it performs automatic normalization, while the Hebb update simply increases the weights without bound creating instability.
The problem is that the NeuroSolutions component utilizes full connectivity between the input and the output, as this is the general case in neurocomputing. However, Freeman specifies the connection of excitactory-to-excitactory or inhibitory-to-inhibitory components ONLY. So a new component is being designed to conform with this design requirement. The adaptation algorithm is still the same, except that it only applies to the connections specified by the interconnection matrix. We already developed the ArbitrarySynapse, so what is needed is to couple it with the learning. We will be testing this modification very shortly.
4.0 Further Work
The work for the next month will progress along the following directions
This will basically cover the objectives of the Phase I proposal.
5.0 Organizational Issues
We have experienced major difficulties in obtained the payments from ONR. The payment of the two first installments was received just May 20, 1997, instead of January as expected. This implied a major commitment of NeuroDimension financial resources and the understanding of all the employees involved. We have already ordered the PC for the simulations and so expect to effectively do the simulations early next month.
References:
[1] - Yao Y., & Freeman W. J. (1990). Model of Biological Pattern Recognition with Spatially Chaotic Dynamics. Neural Networks. Vol. 3. Pp. 153-170.
[2] - Freeman W. J., Yao Y., & Burke B. (1988). Central Pattern Generation and Recognizing in Olfactory Bulb: A Correlation Learning Rule. Neural Networks. Vol. 1. Pp. 277-288.
[3] - De Vries B. and Principe J., (1992). The gamma model: a new neural model for temporal processing,. Neural Networks, vol 5, #4, 565-576, 1992.
[4] - Principe J., deVries B., Oliveira P. (1993). The gamma filter: a new class of IIR filters with restricted feedback. IEEE Trans. Signal Proc., vol 41, #2, 649-656, 1993.
[5] - Oppenheim A. V., & Schafer R. W. Discrete-Time Signal Processing. Prentice Hall