I knew the mechanics but started rambling about the loss surface before actually grounding it in the linear regression setup.
Start by defining the linear regression model and its loss function, then explain how gradient descent iteratively updates the parameters to minimize that loss. Focus on the intuition behind the gradient as the direction of steepest descent and the role of the learning rate, and finally discuss practical considerations like convergence and variants.
Pro tip: Emphasize that while gradient descent is a general optimization algorithm, for linear regression the loss surface is convex, guaranteeing convergence to the global minimum with an appropriate learning rate. Mentioning this shows deeper understanding and can lead to discussions about trade-offs with closed-form solutions.
State the linear regression hypothesis: hθ(x) = θ^T x, and the mean squared error (MSE) loss: J(θ) = (1/2m) Σ (hθ(x_i) - y_i)^2. Explain that the goal is to find θ that minimizes J(θ).
Derive the gradient of the loss with respect to each parameter: ∂J/∂θ_j = (1/m) Σ (hθ(x_i) - y_i) x_i_j. Highlight that the gradient points in the direction of steepest increase, so we move opposite to it.
Present the parameter update: θ_j := θ_j - α * ∂J/∂θ_j, where α is the learning rate. Explain that this step is repeated until convergence (e.g., gradient norm below threshold or max iterations).
Explain that the choice of α is critical: too small leads to slow convergence, too large may cause divergence or oscillation. Mention that for linear regression, the loss is convex, so gradient descent converges to the global minimum given a suitable α.
Briefly cover batch, stochastic, and mini-batch gradient descent, and compare with the closed-form solution (normal equation). Discuss trade-offs: gradient descent scales better to large datasets but requires tuning α and iterations.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.