Bias/Underfit vs Variance/Overfit - Linear Regression

Let’s say we are using Ordinary Least Squares (OLS) Regression as our learning algorithm

The data is given below

We can always find a regression function 𝑓(𝑥) that passes through all the observed points without any error (i.e. 𝛴_{1≤𝑖≤𝑛}[𝑒_𝑖]² = 0). Though this may overfit

We want to find a line equation 𝑓(𝑥) that neither:

• underfits (i.e. high bias)
• overfits (i.e. high variance)

Method 1 - Comparing Line Equations of Different Polynomial Degrees

Click here to expand...

we choose a set of candidate line equations (of increasing polynomial degree):

• 𝑦⁽¹⁾ = 𝜃₀ + 𝜃₁(𝑥)
• 𝑦⁽²⁾ = 𝜃₀ + 𝜃₁(𝑥) + 𝜃₂(𝑥)²
• …
• 𝑦^(𝑑) = 𝜃₀ + 𝜃₁(𝑥) + 𝜃₂(𝑥)² + … + 𝜃_𝑑(𝑥)^𝑑

we split the data into 2 sets:

• training set
• test set / cross-validation set

we train each candidate line equation with the training set. For each candidate we have a model and we can test its performance by calculating its error over the 2 data sets. Then plot it!
bias-variance-polynomial-degree.drawio

we choose the model in the sweet spot region

Method 2 - Comparing Different Regularization Values

Click here to expand...

• single model: 𝘩_𝜃(𝑥) = 𝜃₀ + 𝜃₁(𝑥) + 𝜃₂(𝑥)² + … + 𝜃_𝑑(𝑥)^𝑑
• cost/error function: 𝐽(𝜃) = (1/2𝑛)𝛴_{1≤𝑖≤𝑛}(𝘩_𝜃(𝑥^(𝑖)) - 𝑦^(𝑖))² + (𝜆/2𝑑)𝛴_{1≤𝑗≤𝑑}(𝜃_𝑗)²

try different regularization values 𝜆:

• 𝜆₁ = 0
• 𝜆₂ = 0.01
• 𝜆₃ = 0.02
• 𝜆₄ = 0.04
• 𝜆₅ = 0.08
• …
• 𝜆₁₂ = 10.24

we split the data into 2 sets:

• training set
• test set / cross-validation set

we train each candidate line equation with the training set. For each candidate we have a model and we can test its performance by calculating its error over the 2 data sets. Then plot it!

bias-variance-as-a-function-of-regularization-parameter.drawio

we choose the model in the sweet spot region