LR - Methods Estimating Unknown Regression Coefficients

Method of Least Squares
(Gradient Descent)

• idea: minimizing square error via GRADIENT DESCENT
• need to choose learning rate 𝛼
• need many iterations
• works well when the number of training examples 𝑋 is large

Method of Least Squares
(Projection Matrix - Normal Equation)

• idea: minimizing square error via NORMAL EQUATIONS
• no need to choose learning rate 𝛼
• do not need to iterate
• need to compute (𝑋^𝑇𝑋)^-1𝑋^𝑇 or 𝑉𝐷^-1𝑈^𝑇
• slow if the number of training examples 𝑋 is large because computing the inverse of a matrix is 𝑂(𝑛³)

Maximum Likelihood Estimation

• idea: maximize the likelihood/likelihood function
• For OLS to be mathematically equivalent to MLE, the errors are assumed to be normally distributed and Independent and Identically Distributed (IID)

MAP (Bayesian Linear Regression)

• idea: maximize the posterior

Newton-Raphson (N-R) Technique

• idea: TODO