(1B.18)
We can rewrite the performance surface in terms of its minimum value and w* as
(1B.19)
For the one dimensional case, this equation is the same as Eq.1B.10 (R becomes a scalar equal to the variance of the input). In the space (w1,w2) J is now a paraboloid facing upward. The shape of J is again solely dependent on the input data (through its autocorrelation
function). One can show that:
· the principal axes of the performance surface contours (surfaces of equal
error) correspond to the eigenvectors of the input correlation matrix R, (see Appendix A)
· while the eigenvalues of R give the rate of change of the gradient along the principal axis of the
surface contours of J (Figure 1-16).
Figure 1-16 Contour plots of the performance surface with two weights
The eigenvectors and eigenvalues of the input autocorrelation matrix are all
that matters to understand convergence of the gradient descent in multiple
dimensions. The eigenvectors represent the natural (orthogonal) coordinate system to
study the properties of R. In fact, along these coordinates the convergence of the algorithm can be
studied as a joint adaptation of several (one for each dimension of the space)
unidimensional algorithms. Along each eigenvector direction (the axes of the
ellipsoids) the algorithm behaves just like the one-variable case that we studied in
the beginning of this chapter. The eigenvalue becomes the projection of the
data onto that direction, just as 
in Eq.1B.11 is the projection of the data on the weight direction. But in any other
direction the adaptation is coupled.
The location of the performance surface in weight space depends on both the
input and desired response Eq.1.31. The minimum error also depends on both Eq.1B.18. Multiple regression finds the location of the minimum of a paraboloid placed
in an unknown position in weight space. The input distribution defines the shape of the performance surface. The input distribution and its relation with the desired response
distribution define both the value of the minimum of the error and the location in coefficient space where
that minimum occurs.
Use your browser's back button to return to text.