Greetings from NeuroDimension!
Makers of NeuroSolutions, the Neural Network Simulation Environment.
This issue of the newsletter highlights NeuroSolutions flexibility as a time-series prediction tool. It also announces the release of our new interactive book, Neural and Adaptive Systems: Fundamentals Through Simulations, along with the assortment of tips and techniques that you’ve come to expect from the professionals at NeuroDimension.
In this issue you’ll find:
What’s News?
* Now Available: Neural and Adaptive Systems: Fundamentals Through Simulations
* Preview Chapter One of the Interactive Book Online
Designing Neural Networks
* Memory Design for Temporal Problems
NeuroSolutions Tip Box
* Configuring the Controllers for Dynamic Neural Networks
Did You Know?
* Time-Series Prediction using NeuroSolutions for Excel: Part 1
Customer Spotlight
* Real-Time Current Forecast for Tunnel Element Towing
Note: You are receiving this newsletter because you requested to stay informed concerning new developments at NeuroDimension. If you would like to stop receiving these newsletters, please see the bottom of this newsletter for removal instructions.
=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=
What’s News?
Now Available: Neural and Adaptive Systems: Fundamentals Through Simulations
Our interactive neural network book, Neural and Adaptive Systems: Fundamentals Through Simulations is now complete. It is scheduled for publication by John Wiley & Sons during the last week in November. The cost of the book is $79 (plus shipping) and will be available for shipping during the first week of December.
The book uses NeuroSolutions to actively demonstrate neural and adaptive systems using over 200 interactive examples. The results of our beta testing several months ago were very positive. The book received an average overall rating of 4.4 (5=excellent, 1=poor). Every respondent said that the interactive examples were very helpful in understanding the topics covered in the book. Comments included: "Very good concept, can't get enough!", "It's a book worth reading. It makes the rather complex theory understandable", and "An excellent contribution and a new dimension for workers in this field."
For more information and samples of the interactive book, please visit our web site at: http://www.nd.com/products/nsbook.htm
If you would like to pre-order the book using our secure on-line ordering system, please visit: https://ssl10.pair.com/neurodim/
Preview Chapter One of the Interactive Book Online
If you would like to preview the book, the first chapter is available for viewing online from our web site. The first chapter has 14 sections and covers such topics as linear models, regression for multiple variables, Newton’s method and more. This online version does not include the interactive examples.
To view the html version of chapter one, please visit:
The interactive examples of the book are fundamental to the presentation and understanding of the topics. The full Evaluation version of NeuroSolutions has been updated to include the publication version of chapter one of the book, complete with interactive examples.
To download the full Evaluation version of NeuroSolutions with chapter one and examples (11.2 MB), please visit:
http://www.nd.com/download/full_form.htm
=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=
Designing Neural Networks
This Month: Memory Design for Temporal Problems
Neural networks are frequently used to predict data that occurs over a period of time. For this type of problem, the most recent values can be used to predict future values. This is an example of a temporal problem, or modeling a value through time.
To create neural networks to solve temporal/dynamic problems we must include memory in our architecture. The most common method of adding memory is to include a tap-delay line to the network. The tap-delay line is a single-input multiple-output device that outputs not only the current input but the N past inputs as well. By looking at the current and past inputs of the system, the network can extract temporal information from the signal.
The tap-delay line (the TDNNAxon in NeuroSolutions) is restricted in its functionality. To explain this, we must discuss memory depth and resolution. Memory depth is how far into the past you have information about the signal. Memory resolution is the amount of information you have about each past input. A tap-delay line with N taps has a memory depth of N-1 (current input plus N-1 past inputs) and a memory resolution of 1 (every past sample is perfectly represented). The resolution is always 1 and to get more depth you must add more taps to the memory (and more weights to the network).
There are times, however, when you may need a large memory depth (things in the distant past have a large effect on the current output) but not necessarily perfect representation. As an example, think of a memory device that has 3 outputs, where the first one is the average of the samples at time t, t-1, and t-2, the second tap produces the average of the samples at t-3, t-4, and t-5, and the third tap produces the average of the samples at t-6, t-7, and t-8. This memory device has a memory depth of 8 (with only 3 taps) but a resolution of only 1/3. If you need to look 8 samples in the past but don’t need the exact values of each sample, this type of memory device may be significantly better than a 9-tap-delay line because it provides the necessary information but with 3 times less weights (3 taps instead of 9).
The gamma memory and Laguarre memory are similar to this averaging filter. They can adaptively trade-off memory resolution for memory depth. More importantly, they can be trained to select the best depth/resolution setting to solve the problem. Also, they function identically to a tap-delay line when their mu parameter is set to 1, thus they subsume the tap-delay line and should not produce results worse than the tap-delay line (if the tap-delay line is the best architecture, they should train to that architecture).
In summary, the gamma and Laguarre memories provide an adaptable memory structure that at one extreme emulates the tap-delay line. The Laguarre memory is theoretically more complex, but practically is no more difficult to use than the gamma memory. Since the Laguarre memory can better represent information from the distant past, we suggest the Laguarre memory as the memory of choice.
=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=
NeuroSolutions Tip Box
This Month: Configuring the Controllers for Dynamic Neural Networks
When using some variant of the tap-delay neural network as described above, you will often use trajectory learning (see the "Designing Neural Networks Section" of the May newsletter -- http://www.nd.com/newsletter/051899.htm). The NeuralWizard will configure the DynamicControl and BackDynamicControl components automatically for trajectory learning, but it is good to know the parameter settings available to you in order to fine tune the network for your particular problem.
Dynamic Control
Samples/Exemplar - The number of individual pieces of temporal information that comprise a complete pattern of samples. For example, suppose you are working on a voice recognition application and you have 1-second voice recordings for 100 different words. Each recorded second of sound (i.e., one word) is represented by 8000 values resulting in a total of 800,000 samples of data. Therefore, you would want to configure the controller for 8000 samples/exemplar and 100 exemplars/epoch.
Zero State Between Exemplars - When this switch is turned on and an exemplar has completed, all components that store internal states (e.g., MemoryAxons, delayed Synapses) will have those states set to zero before the next exemplar. For example, a TDNNAxon would have all of its memory taps set to 0 after each exemplar.
Zero State Between Epochs – Same as above except the states are zeroed after each epoch instead of exemplar.
BackDynamic Control
Samples/Exemplar – The number of samples that the network is backpropagated after each exemplar in order to compute the gradients and sensitivities. This value can range anywhere between 1 and the number of Samples/Exemplar in the DynamicControl. Typically, this is set to the number of Samples/Exemplar minus the number of memory taps. This allows the network to train on all of the available information while skipping the samples during which the taps did not have useful information.
=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=
Did You Know?
This Month: Time-Series Prediction using NeuroSolutions for Excel: Part 1
Assume you have a simple time-series prediction problem where the last three periods of the time series are all that is necessary to predict the value of the time series one period in advance. There are two ways to set up a problem like this for use with NeuroSolutions for Excel -- one method uses a multilayer perceptron (MLP), the other involves using a focused time delay neural network to perform the same task with fewer steps.
Method 1: Using a multilayer perceptron (MLP)
1. Starting with a single labeled column of data (the time series you want to predict), copy this column of data (select the data only, not the whole column or the label) and paste it into the two adjacent columns to the right. Start one cell lower with each pasting operation. Tag these three columns as inputs.
2. Copy the original column of data again and paste it in the first blank column to the right, but this time start one cell higher than the original column. Tag this new column as desired.
3. Add labels to the three new columns of data that you created.
4. Delete all of the rows that do not have complete data (rows 2, 4 and the last 3 rows)
5. Tag the rows of data as training, cross validation, and/or testing.
6. Create an MLP then train and test the network as usual.
Here is an example of what the processed data will look like when starting with a single column of data consisting of the numbers 1 through 7 (the time series).
Processed Data
x(0) x(-1) x(-2) x(1)
----------------------------------------------------------
3 2 1 4
4 3 2 5
5 4 3 6
6 5 4 7
This article has shown you how to prepare data for time-series prediction for use with a multilayer perceptron neural network. The next newsletter will discuss the second method, which involves using a focused time delay neural network to perform the same time-series prediction task.
=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=
Customer Spotlight
This Month: Real-Time Current Forecast for Tunnel Element Towing
Vladan Babovic
Danish Hydraulic Institute
The construction of the fixed link across Øresund — following the 1991 agreement between the Swedish and Danish governments — is now almost completed. The fixed link consists of a combined twin track railway and a four-lane motorway. The total link length will extend just under 16 km between Kastrup on the Danish coast and Lernacken on the Swedish coast. One of the elements of the link is an immersed 3510-meter long tunnel under the Drogden navigation channel.
The immersed part of the tunnel consists of 20 elements, each approximately 175 meter long, resulting in a total immersed tunnel length of 3510 m. The outer cross-sectional dimensions are 8.6 m by 38.8 m, enclosing two railway tubes, two motorway tubes, and a central escape and installation gallery. The elements are placed in a pre-dredged trench, and founded on bedding sand.
All 20 tunnel elements are fabricated in a purpose-built casting yard at the Nordhavn area of Copenhagen harbour. The transportation distance to the tunnel site is 20 km. The towing configuration is with two leading tugs and two assisting tugs, i.e. one at each corner of the element. Figure above indicates locations of casting yard, tunnel site and towing route. The total duration of a towing and element immersion is expected to last up to 36 hours.
The planning of the transportation and installation of tunnel elements requires a wealth of knowledge about the current conditions in Øresund. In order to assist in a planning of such a critical operation Danish Hydraulic Institute developed a real-time forecast system producing predictions of speed and direction of currents at the location called Nordre Røse (situated near tunnel site) as well as water levels at Nordre Røse and Dragør.
NOTE: This is just an abstract of the application summary. The entire summary is available at:
http://www.nd.com/application%20summaries/appsum-forecast.htm
Want to have your solutions spotlighted? We strongly encourage our customers to send their 1-2 page application summaries to submissions@nd.com so that we may post them on our web site at: http://www.nd.com/applicationsum.htm. In each newsletter, we’ll spotlight a new solution and include a link for people to get more information.
=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=
Comments or Suggestions?
We appreciate your feedback! Please send us your comments or suggestions concerning this newsletter, our web site, or part of the NeuroDimension product line. Write to us at: feedback@nd.com
Have questions about NeuroDimension products or training services? Send your questions to: info@nd.com
This issue and previous issues of this newsletter are available on the NeuroDimension web site at: http://www.nd.com/mailinglist.htm
If you would prefer not to receive these newsletters or subsequent product updates from NeuroDimension, please reply to this letter with the subject heading changed to the word REMOVE.
Thank you again for your support of NeuroDimension products!