LASSO Regression (Least Absolute Shrinkage and Selection Operator)

LASSO - Estimator

The LASSO estimator 𝛽ˆ_{𝐿𝐴𝑆𝑆𝑂} is obtained by minimizing:

• (𝑦 - 𝑋𝛽)^T(𝑦 - 𝑋𝛽)

subject to:

• ${\sum }_{i=1}^p|{\beta }_i|\le t\text{ for some }t>0$

or equivalently:

• ${\sum }_{i=1}^p||{\beta }_i|{|}_1\le t\text{ for some }t>0$

The value of 𝑡 determines the degree of regularization.

The problem written in Lagrangian form is to minimize:

• $I_{\lambda }(\beta )=(y-X\beta {)}^T(y-X\beta )+\lambda ||\beta |{|}_1$

LASSO - Interpretation of Values of 𝜆

• if 𝜆=0, then we recover the usual unbiased estimator for 𝛽
• if 𝜆=∞, then 𝛽=0

As 𝜆 increases:

• the more estimators 𝛽_𝑖 = 0
• the bias increases
• the variance decreases

Subpages

• Ridge Regression vs LASSO Regression