Full search page →

PR 03 · Complexity versus shrinkage

IntermediatePR 03 · Linear Models for Regression and ClassificationInteractive plot

Description

Fit polynomial basis functions with ridge regression and separate training error from recovery of the underlying function.

Try this: At degree seven, increase regularization. Does the lowest training error also give the lowest underlying-function error?

Predict first

Set λ = 0 and raise the degree from 5 to 6. Training error falls (0.0285 to 0.0239). What happens to the error against the true function?

Hints

The chapter section that uses this lab:

Explanation

Answer the prediction first, or open the explanation when you are ready.

Show the explanation

It rises from about 0.0005 to about 0.0063. The extra flexibility fits the noise in the 12 training points, which lowers the training error and moves the curve away from the true function.

Lab · PR 03 · Complexity versus shrinkage

Predict first

Set λ = 0 and raise the degree from 5 to 6. Training error falls (0.0285 to 0.0239). What happens to the error against the true function?

Training mean squared error
0.0538
True-function grid error
0.029
Squared coefficient norm
5.403

Changing λ alters the fitted coefficients, not the observations. Large penalties can remove noise-fitting but also suppress the curvature needed to represent the signal.

Try this: At degree seven, increase regularization. Does the lowest training error also give the lowest underlying-function error?

The blue fit solves sum of squared residuals + λ times the squared non-intercept coefficients. Twelve fixed synthetic points have deterministic perturbations. The dashed curve is sin(πx); its grid error is available only because this is a known synthetic problem. Polynomial features are linear in their coefficients.

Method sources

Open in the Playground →