I knew the OLS objective well enough, minimizing sum of squared residuals, and got through the matrix form (beta = (X'X)^-1 X'y) without too much fumbling.
Start by defining the least squares problem as minimizing the sum of squared residuals, then derive the normal equations to get the closed-form solution. Next, list the key assumptions (linearity, independence, homoscedasticity, normality) and explain how ridge regression adds an L2 penalty to address multicollinearity and overfitting, contrasting it with OLS.
Pro tip: Mention that while the closed-form solution is elegant, in practice for large datasets you'd use gradient descent or QR decomposition for numerical stability, showing you understand both theory and implementation trade-offs.
State that least squares minimizes the sum of squared residuals between observed and predicted values, typically in linear regression.
Set up the objective as ||y - Xβ||², take the gradient with respect to β, set it to zero, and solve to get β = (XᵀX)⁻¹Xᵀy.
List assumptions: linear relationship, independent errors, constant variance (homoscedasticity), and normally distributed errors for inference.
Explain that ridge adds an L2 penalty term λ||β||² to the objective, modifying the solution to β = (XᵀX + λI)⁻¹Xᵀy, which shrinks coefficients and handles multicollinearity.
Contrast OLS and ridge: OLS is unbiased but high variance; ridge introduces bias but reduces variance, improving prediction. Mention practical implications like choosing λ via cross-validation.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.