Least Squares Derivation

In Eq.1.4 we substitute the value of the error given by Eq. 1.3 and take the derivative expressed by Eq.1.5 to obtain

(1B.1)

Note that to simplify the notation we omit the limits of the variable in the sum. We do this throughout the text when no confusion arises. Operating we get further

(1B.2)

The set of Eq. 1B.2 is called the normal equations. The solution of this set of equations is

(1B.3)

which provides the coefficients for the regression line of d on x. The summations run over the input-output data pairs. Eq. 1B.2 is solved by computing the value of b from the first equation and substituting it in the second equation to obtain w as a function of x and d. Then the value of w is substituted in the first equation to finally obtain b as a function of x and d (variable elimination). It is easy to prove that the regression line passes through the point

which is called the centroid of the observations. The denominator of the slope parameter of w and b is the corrected (for the mean) sum of squares of the input.

Use your browser's back button to return to text.