1.2 Linear Models

From the simple observation of Figure 1-3, it is obvious that the relationship between the two variables x and d is complex, if we assume that no noise is present. However, there is an approximately linear trend in the data. The deviation from a straight line could be produced by noise, and underlying the apparent complexity could be a very simple (possibly linear) relationship between x and d, that is

(1.1)

Figure 1-3 Plot of x versus d

or more specifically,

(1.2)

where i is the instantaneous error that is added to yi (the linearly fitted value), w is the line slope, and b is the d axis intercept (or bias). Assuming a linear relationship between x and d has the appeal of simplicity. The data-fitting problem can be solved by a linear system with only two free parameters, the slope w and the bias b.

Figure 1-4 Linear regression PE

The system of Figure 1-4 is called the linear processing element (PE), or Adaline (for adaptive linear element), and it is very simple. It is built from two multipliers and one adder. The multiplier w scales the input, and the multiplier b is a bias, which can also be thought of as an extra input connected to the value +1.The parameters (b, w) have different functions in the solution. We will be particularly interested in studying the dependence of the solution on the parameter(s) that multiply the input xi.

NEUROSOLUTIONS EXAMPLE 1.1

The Linear Processing Element in NeuroSolutions

The goal of this book is to demonstrate as many concepts as possible through demonstrations and simulations. NeuroSolutions is a very powerful neural network/adaptive system design and simulation package that we will use for the demonstrations. We highly recommend that you read through the NeuroSolutions tutorial in Appendix B. NeuroSolutions constructs adaptive systems in a Lego style, that is component by component. The components are chosen from palettes, selected with the mouse, and dropped into the large window called the breadboard. This object-oriented methodology allows for the simple creation of adaptive systems by simply dragging and dropping components, connecting them, and then adjusting their parameters. Particularly in the early chapters we will automatically create the adaptive systems for you through a set of macros. This will shield you from the details of NeuroSolutions until you have a better grasp of the fundamentals of adaptive systems and the use of NeuroSolutions.

In this first example, we introduce a few simple components. The first component required in any simulation is an input component, which belongs to the Axon family. Its function is to receive data from the computer file system or from signal generators within the package. In this case, we will add a file input component to the input Axon to read in the data from Table 1-1. The linear PE shown in Figure 1-4 can be constructed with a Synapse and a Bias Axon. The Synapse implements a sum of products and the Bias Axon adds the bias. The output of such system is exactly Eq. 1.1. The Controller manages the system and controls the firing of data through the system. Since Table 1-1 has 12 data points, the Controller is configured to send 12 points through the system.

The purpose of this example is to display the output of the linear PE, which is a line, and to modify its location in the space by entering different slope and bias values. To display the input and regression line, we use the Data Storage component (stores 12 samples) and the Scatter Plot component. The Scatter Plot component allows us to plot the input (x axis) versus the system response (y axis). We also add two Matrix Editor boxes to allow you to change the values of the two parameters: the weight (slope) and bias (y-intercept). After changing these parameters, you use the Control Palette to run the network.

NeuroSolutions Control Palette

The Run button (the green triangle) tells the Controller to send the data through the network. The other buttons are not important now, but will be used and explained later. Now run the NeuroSolutions example by clicking on the yellow NeuroSolutions icon below. It will walk you through the creation of the breadboard and allow you to see how the regression line changes as you change the weight (slope) and bias (y intercept).

NeuroSolutions Example

(Reminder, click on the yellow icon to start the example.)

Go to next section