Subject: NeuroDimension News and Tips V1 #6

Greetings from NeuroDimension!

Makers of NeuroSolutions, the Neural Network Simulation Environment.

NeuroDimension welcomes our academic customers back from their summer breaks. This issue of the newsletter highlights tips and techniques that will help you to get the most out of NeuroSolutions from the professionals at ND.

And just a reminder, in addition to NeuroSolutions being a flexible research and development environment, its breadboard-style interface also makes an excellent teaching tool for both neural networks and adaptive signal processing. Look for our new book "Neural And Adaptive Systems: Fundamentals Through Simulations" later this fall! For more information, see http://www.nd.com/products/nsbook.htm

In this issue you’ll find:

What’s News?

Designing Neural Networks

NeuroSolutions Tip Box

Did You Know?

Customer Spotlight

Note: You are receiving this newsletter because you requested to stay informed concerning new developments at NeuroDimension. If you would like to stop receiving these newsletters, please see the bottom of this newsletter for removal instructions.

What’s News?

New NeuroSolutions Course Offering

NeuroDimension is offering another set of courses November 8-12, 1999 in Orlando, Florida. We continue to receive rave reviews from our attendees, in fact everyone has said they would recommend our courses to friends. We average a score of 4.6 of 5 when attendees are asked to rate the overall quality of the course. Our overlapping course format allows both novice and advanced users to find a suitable course. Offered courses include: "Introduction to Neural Networks and NeuroSolutions", "Advanced Neural Networks and NeuroSolutions", and "NeuroSolutions for Developers".

Remember to sign-up early and receive a 10% early registration discount.

For details on the November offerings, see http://www.nd.com/course/november.htm

For general ND course information, see http://www.nd.com/course/

Year 2000 Compliance in NeuroDimension Products

The Y2K bug has many companies scrambling to get their software ready for the new year. This is one thing you don't need to worry about with any of our products. All NeuroDimension products have been tested and are guaranteed to be Year 2000 Compliant. If your business or organization requires a written certificate of compliance, just send e-mail to marlo@nd.com or visit http://www.nd.com/y2k.htm

Designing Neural Networks

This Month: Batch, Online, and Custom Weight Updates

In gradient descent learning, the goal is to estimate the gradient of the performance surface and follow the steepest slope to the bottom. The performance surface is the plot of the mean square error versus the network weights, so gradient descent searches for the weights that give the minimum mean square error – e.g. the best performing network. Since calculating the actual gradient is very time consuming, all of the typical neural network training methods use an estimate of the gradient. The theory is that if you can quickly compute many noisy gradients, then on average you will still move towards the bottom of the performance surface. The trade-off is whether it is more efficient to make a few very accurate gradient calculations or many, many inaccurate gradient calculations.

In NeuroSolutions, the "BackStaticControl" component (the red dials) controls the error calculations and gradient descent learning. On the "BackStaticControl" Inspector, on the "Backpropagation" page, there are three radio buttons labeled "on-line", "batch", and "custom". These selections determine how much time is spent calculating the gradient before the weights of the system are updated. "On-Line" learning indicates that the weights will be updated every exemplar. This is the true Least Mean Squares (LMS) algorithm and is a very fast but noisy estimate of the gradient. "Batch" learning averages each exemplar-by-exemplar gradient over the entire epoch and then applies the weight update once at the end of the epoch. By averaging the noisy gradients, "batch" learning obtains a much better approximation to the real gradient.

In general, batch updates are normally best for data sets with relatively few exemplars (e.g. 500 or less). When you have a large number of exemplars, however, the batch update suffers from two problems. First, when you average too many gradients you may be masking or losing too much information from the individual gradients. Second, you spend a great deal of time processing the data and not enough time modifying the weights. With large data sets, on-line learning normally trains much faster. On-line learning, however, sometimes suffers from a problem called "tracking". Tracking is when a network trains too quickly on the local information and doesn’t ever capture the more global information contained in the entire data set. For example, if the first half of your data has many instances of "class 1" and the second half has many instances of "class 2", then a network that is tracking will repeatedly learn about class 1, then forget class 1 and learn about class 2, without ever learning how to discriminate between them. For large data sets, the best answer is a "custom" weight update. Click the "custom" radio button and then enter a value for "exemplars per update". We suggest approximately 100 exemplars per update, which provides a reasonable tradeoff between batch and online.

NeuroSolutions Tip Box

This Month: Graphically Displaying the Output and the Desired Signal

As you have probably seen from the "Time Lagged Recurrent" demo, NeuroSolutions has the capability to graphically display the network output and the desired output on the same MegaScope while the network is training. Configuring the probes to do this on your own network does require a few steps. We have written a simple macro that will perform these steps automatically. It works with standard (NeuralWizard generated) breadboards that use static backpropagation. This macro can be downloaded from: http://www.nd.com/newsletter/OutputvsDesired.nsm

Did You Know?

This Month: More on Parameter Optimization using NeuroSolutions for Excel

The May issue of the newsletter introduced using the "Vary a Parameter" batch in NeuroSolutions for Excel for optimizing neural network parameters. Many users don’t realize that this optimization can be performed on just about any neural network parameter, not just the parameters available within the "Component.Action" combo box. To optimize other parameters, simply type the component and action to perform directly into the Component.Action combo box.

For example, suppose you would like to find a good value of "Beta" to use for a LinearAxon that is serving as the output axon of your network. Recall that the formula for the output of a processing element within a LinearAxon is: Beta * Input + Bias, so varying the Beta will change the slope of the linear transfer function. To vary this parameter, you would enter "outputAxon.setBeta" into the "Component.Action" combo box where "outputAxon" is the name of the component to perform the action on and "setBeta" is the action to perform on the component.

Where did this parameter name come from? The name of the component ("outputAxon") can be found on the "Engine" tab within the components inspector in NeuroSolutions. The action ("setBeta") is described in the NeuroSolutions help. In this case, see "LinearAxon Macro Actions". For a list of actions that can be performed on other NeuroSolutions components, see the NeuroSolutions help.

Customer Spotlight

This Month: NeuroDimension, Inc.

NeuroDimension, Inc. has recently completed work on a research project funded by the National Science Foundation. The goal of this project was to research and develop an adaptive inverse control architecture to control a medical ventilator using NeuroSolutions. In this project we developed some very interesting technology, both in terms of the NeuroSolutions development environment and in terms of adaptive inverse control.

A summary of this research is available on our web site. It highlights the flexibility of NeuroSolutions in developing custom architectures and the power of OLE automation for real-time control. It is also an excellent example of using neural networks for control in general. The research summary can be found at http://www.nd.com/ventilator/

Want to have your solutions spotlighted? We strongly encourage our customers to send their 1-2 page application summaries to submissions@nd.com so that we may post them on our web site at: http://www.nd.com/applicationsum.htm. In each newsletter, we’ll spotlight a new solution and include a link for people to get more information.

Comments or Suggestions?

We appreciate your feedback! Please send us your comments or suggestions concerning this newsletter, our web site, or part of the NeuroDimension product line. Write to us at: feedback@nd.com

Have questions about NeuroDimension products or training services? Send your questions to: info@nd.com

This issue and previous issues of this newsletter are available on the NeuroDimension web site at: http://www.nd.com/mailinglist.htm

If you would prefer not to receive these newsletters or subsequent product updates from NeuroDimension, please reply to this letter with the subject heading changed to the word REMOVE.

Thank you again for your support of NeuroDimension products!