Final Report For Contract

ONR- N00014-97-C-0049

"Biological Extensions to NeuroSolutions"

for the period

May 1, 1997 - June 30, 1997

Prepared by

Curt Lefebvre

NeuroDimension, Incorporated

1800 North Main Street, D4

Gainesville, FL 32609

July 2, 1997

1. Introduction

This report is divided into two parts: a report of the work conducted with presentation of results and the future work. We also present in the Appendices the code developed during the research and a step-by-step implementation of the Freeman's model in NeuroSolutions.

2. Synopsis of the project

The goal of this project was to study the feasibility of extending NeuroSolutions, an artificial neural network software package being commercialized by NeuroDimension Incorporated, with biologically motivated components. Biological neural networks are believed to embed principles far superior to the ones presently utilized in artificial neural networks. Therefore, it is extremely important to find out if it is possible to encode present biological motivated models in commercial neural network packages such that researchers can experiment and fully utilize them for the engineering tasks of classification and temporal processing.

The first goal was to find out which biological motivated neural models are available and applicable to engineering applications. This was done during the early phase of the research (see Report I). We decided to investigate Freeman's model of the olfactory system as the model to be implemented in NeuroSolutions because:

• Freeman's model is well rooted in biology
• Freeman's model is a computational model
• Freeman's model has been applied to pattern recognition showing promise of outperforming other neural models.

The challenges to implement Freeman's model in NeuroSolutions were the following (see Reports II):

• Freeman's model is a dynamic model (i.e. defined by differential equations), normally implemented in analog hardware or digitally simulated using Runge-Kutta integration. So we had to find out new ways to map the differential equations to digital implementations to turn it practical.
• Freeman's model is a distributed model of coupled oscillators, unlike any other in neural networks. So we had to address which are the individual components and how to implement them in the neural classes already implemented in NeuroSolutions.
• Freeman's model required novel components which needed to be programmed in NeuroSolutions.
• Freeman's model also required a new user interface due to the massive architecture of the model.

We are pleased to report that the feasibility study was successfully completed, and that we now have working inside NeuroSolutions the basic Freeman's model of the olfactory bulb (the KII model). The full Freeman's model (the KIII model) is a larger interconnection of the same components utilized for the olfactory bulb, and was not simulated during Phase I due to lack of time. The basic results of this project are the following (see Reports II and III):

• We have proposed a new implementation of the Freeman PE as a dynamic network based on a gamma processing element followed by a nonlinearity (Report II). This circumvents the need for Runge-Kutta integration.
• The Freeman PE can be implemented efficiently in digital computers, requiring a computational bandwidth far less than the traditional integration (Report II).
• We also have demonstrated that the waveforms produced by the NeuroSolutions implementation correspond to the digital implementation using Runge-Kutta integration in MATLAB. This justifies our claim that Freeman's model can be implemented using a network of components instead of solving the difference equations using Runge-Kutta integration with the advantages of efficiency and a much more friendly graphical user interface (Report II and III).
• We have specified the minimal number of NeuroSolutions components that needed to be developed to implement the Freeman's PE, as well as their inclusion in the package's families of neural components (Report III). We also addressed how Freeman's PE should be encapsulated in an icon (Report III).
• We have developed the NeuroSolutions code to implement such enhancements as we will discuss next.

3. NeuroSolutions Implementation of Freeman’s dynamic PE

3.1.Implementing the Freeman PE

The KII model is an eighth order oscillator created from 4 K0 cells (Figure 1) with dynamics described by a linear part given by Eq.1

Equation 1

where a and b are constants, and x(t) represents the membrane voltage, followed by a nonlinear part y=Q(x) where

Equation 2

is based on the gamma neural model followed by a nonlinearity Q(x) and a special interconnection. Our proposal to implement this dynamical system is to match the dynamics of Eq. 1 with a three tap gamma memory with the following coefficients:

NeuroSolutions already implemented the gamma memory and the other components required to simulate this model utilizing the bilinear transform to map continuous to discrete time (see Report II).

3.2. Creating a new nonlinearity

The nonlinearity proposed by Freeman is shown in Eq. 2. We created a DLL (Dynamic Link Library) to implement this nonlinearity and include it in NeuroSolutions directly. The code developed is:

// Dynamic link library implementation of NeuroSolutions TanhAxon component

#include "NSDLL.h"

/************************************************/

/* Macro to access the PE layer in matrix form */

#define data(i,j) data[j+i*cols]

/***********************************/

/* Forward activation of component */

__declspec(dllexport) void performLinearAxon(

DLLData *instance, // Pointer to instance data (may be NULL)

NSFloat *data, // Pointer to the layer of processing elements (PEs)

int rows, // Number of rows of PEs in the layer

int cols, // Number of columns of PEs in the layer

NSFloat *bias, // Pointer to the layer's bias vector, one for each PE

NSFloat beta // Slope gain scalar, same for all PEs

)

{

int i, length=rows*cols;

for (i=0; i<length; i++)

{

if (data[i]>(NSFloat)log(1.0-bias[i]*log(1.0+1.0/bias[i])) )

data[i]=bias[i]*(NSFloat)(1.0-exp(1.0/bias[i]*(NSFloat)(exp(data[i])-1.0)));

else

data[i]=(NSFloat)(-1.0);

}

}

/******************************************/

/* Management of instance data (OPTIONAL) */

/*

__declspec(dllexport) DLLData *allocLinearAxon(

DLLData *oldInstance, // Pointer to the last instance if reallocating

int rows, // Number of rows of PEs in the layer

int cols // Number of columns of PEs in the layer

)

{

DLLData *instance = allocDLLInstance(oldInstance);

return instance;

}

__declspec(dllexport) void freeLinearAxon(DLLData *instance)

{

freeDLLInstance(instance);

}

*/

3.3 Special interconnection to create the KII model

Freeman proposes a special interconnection between four PEs to simulate the dynamics among excitactory (e) and inhibitory (i) cell assemblies.

Figure 1. The 4 dynamic blocks that constitute the KII model

He estimates the weights among these components from in vitro and in vivo measurements. The weights that he suggests were utilized in our studies. The values implemented are as follow:

The NeuroSolutions breadboard that implements this model is shown in Figure 2

Figure 2. NeuroSolutions Implementation of Freeman's PE.

It is composed of an Axon connected by an ArbitrarySynapse to a GammaAxon. This constitutes the input to the PE. The GammaAxon implements a dynamical system with 3 taps, and it is fed to a SigmoidAxon. This arrangement implements the second order difference equation followed by the nonlinearity. Each component in the breadboard is a vector of 4 identical individual components. Each arrangement (represented by a circle in Figure 1) is partially connected to the others. This is implemented by the ArbitrarySynapse at the top of Figure 2. The weights are the ones presented in Table II. This constitutes the KII model which we call also the Freeman's PE.

When a square wave input is applied to the Freeman's PE the system should oscillate when the input is positive. The simulation is shown in Figure 3.

Figure 3. Output of the Freeman's PE to a qaure wave. The figure shows the transition from high to low to show the switching behavior of the implementation.

This breadboard shows the waveforms created by each one the PEs when the input is high (first trace) and the transition to low. The system is oscillating when the input is active and transits to silence when the input is zero. This is the expected behavior quoted in the literature by Freeman. The oscillatory behavior obtained in NeuroSolutions was replicated in MATLAB using the Runge-Kutta integration of the original Freeman's equations with the same parameters. So we conclude that the NeuroSolutions implementation and the approximation using the gamma memory is appropriate for digital siulations.

3.4. The model of the olfactory bulb

The olfactory bulb model is built from many KII PEs. Freeman states that less than 64 such oscillators are insufficient to capture the emergent properties seen in biology. Here we created a breadboard with only 2 KII PEs to illustrate the dynamic coupling (Figure 4). In fact if we just excite one of the KII, the other follows oscillation within one or two cycles. This is also expected from the literature.

Figure 4. A set of 2 Freeman PEs in NeuroSolutions. The arbitrary synapses in the middle of the figure perfrom the coupling between the individual PEs.

In this case the coupling coefficients are 0.05 for the excitactory PE and -0.1 for the inhibitory PE. Modulation of the excitactory PE coupling allows the spatial propagation of the excitactory wave, which is also referred in the literature. These ArbitraySynapses are the ones that are adapted using Hebbian learning. So we conclude that our simulation embeds the behavior discussed in Freeman's papers.

3.5 Collapsing the breadboard

The fundamental problem found in our simulations is the huge amount of components necessary to implement the primary building block of the simulations. Figure 4 shows only two KII models interconnected and the screen is already full of components.

We have created a new NeuroSolutions component that encapsulates the KII PE into a single icon, which is a vector of KII PEs (Figure 5). The user can therefore select the number of individual KII PEs from 1 to 100 in the Inspector, and only one icon is shown. This breadboard implements exactly the same networks of Figure 4. The coupling parameters appear now on the ArbitrarySynapse on top. This is the only part of th emodel that is adapted according to Freeman.

This simplicity in the user interface means that the parameters have to be entered in a file instead of the inspector due to the large number. However, the parameters can be read from a file for efficiency. We present in the Appendix the code specially developed to collapse the KII PE and to create the new component.

Figure 5. A network of 2 Freeman PEs (equivalent to Figure 4). The user specifies the number of components in the Inspector, but the parameters have to be entered by editing the parameter file or loaded directly.

3.6. Asynchronous versus synchronous firing

The issue of simulating in digital hardware distributed systems with feedback is a nontrivial one. The problem is how to chain the computations in the serial computer. The most widely used method (because it is the most efficient) is to perform the computations as vector-matrix operations which automatically synchronizes the computations at the time of the products. Within the Freeman PE this provides accurate simulation of the analog computations since the regime is oscillatory. We showed above that the output mimics the results of the Runge-Kutta integration.

A different story occurs when we interconnect together several Freeman PEs, since now there may be delays in the interconnection and each system has a different input. We believe that this point must be carefully studied. We created an ArbitrarySynapse that simulates asynchronous firing by picking at random which is the next input that it will work with. Effectively the asynchronous firing slows down the processing because the simulator is doing vector matrix multiplications with all the components of the vector equal to zero except one. Nevertheless this scheme allows for the comparison of the behavior of the outputs for both synchronous and asynchronous firing.

We developed the code for the ArbitrarySynapse with asynchronous firing (see Appendix II). We did the functional testing (i.e. the component runs without logic errors inside NeuroSolutions) but due to lack of time we did not tested it thoroughly. The component is also prepared to be integrated with Oja's rule to be trained with Hebbian learning.

4.0 Further Work

The Phase I grant showed the feasibility of implementing in the NeuroSolutions software package Freeman's model, which is one of the most intriguing biological realistic neural network models. We have undertaken the preliminary studies to discretize the nonlinear dynamical system into a network topology which is much more efficient than Runge-Kutta integration. This scheme also enables the use of an icon based interface where the user can easily specify the topology and the parameters.

We have also shown that the KII model implemented in NeuroSolutions provides outputs equivalent to Runge-Kutta integration with the same parameters. We also started the necessary steps to make the user interface more versatile to implement in NeuroSolutions biological realistic neural models.

We therefore consider that the goals of the Phase I grant have been met, and that the project provides an excellent start to develop a full blown NeuroSolutions implementation of the olfactory system and test its quality and robustness for pattern recognition and temporal processing.

• The steps that have to be undertaken in the Phase II can be clearly enumerated:
• Test the asynchronous firing in the KIII model
• Implement Hebbian learning
• Implement the dispersive delays
• Implement the full olfactory system model.
• Test the results in practical problems
• Study methods to speed up the computations
• Commercialize Freeman's model in NeuroSolutions as a biological add on.
• We plan to propose this work plan for a Phase II grant after discussions with our ONR program monitor.

References:

[1] - Yao Y., & Freeman W. J. (1990). Model of Biological Pattern Recognition with Spatially Chaotic Dynamics. Neural Networks. Vol. 3. Pp. 153-170.

[2] - Freeman W. J., Yao Y., & Burke B. (1988). Central Pattern Generation and Recognizing in Olfactory Bulb: A Correlation Learning Rule. Neural Networks. Vol. 1. Pp. 277-288.

[3] - De Vries B. and Principe J., (1992). The gamma model: a new neural model for temporal processing,. Neural Networks, vol 5, #4, 565-576, 1992.

[4] - Principe J., deVries B., Oliveira P. (1993). The gamma filter: a new class of IIR filters with restricted feedback. IEEE Trans. Signal Proc., vol 41, #2, 649-656, 1993.

[5] - Oppenheim A. V., & Schafer R. W. Discrete-Time Signal Processing. Prentice Hall, 1985.

NeuroSolutions Home Page