I knew the basics: repeatedly multiply by the matrix and normalize.
Start by defining the power method and its purpose: finding the dominant eigenvalue and corresponding eigenvector of a matrix. Then explain the iterative algorithm step-by-step, and finally justify convergence using the eigenvalue decomposition and the dominance of the largest eigenvalue. Emphasize practical considerations like normalization and convergence criteria.
Pro tip: Mention that the convergence rate depends on the ratio of the second-largest to largest eigenvalue magnitude, and that the method fails if the dominant eigenvalue is not unique or if the initial vector is orthogonal to the dominant eigenvector. This shows deeper understanding and awareness of edge cases.
State that the power method finds the dominant eigenvalue (largest magnitude) and its eigenvector for a diagonalizable matrix with a unique dominant eigenvalue. Assume the matrix has a complete set of eigenvectors and the initial vector has a nonzero component along the dominant eigenvector.
Explain the iterative process: start with a random vector, repeatedly multiply by the matrix, and normalize the result to avoid overflow/underflow. The vector converges to the dominant eigenvector, and the Rayleigh quotient or norm ratio gives the eigenvalue.
Show that after k iterations, the vector is proportional to A^k times the initial vector. Express the initial vector in the eigenbasis, and note that the component along the dominant eigenvector grows relative to others by a factor of (λ2/λ1)^k, which tends to zero if |λ1| > |λ2|.
Mention normalization, convergence criteria (e.g., change in eigenvector or eigenvalue below tolerance), and limitations (e.g., slow convergence if eigenvalues are close, failure if initial vector is orthogonal to dominant eigenvector).
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Start by explaining that standard PCA reduces to a tractable eigenvector problem because the objective is convex and the solution is given by the spectral decomposition of the covariance matrix. Then highlight that adding sparsity constraints (e.g., L0 or L1 penalties) breaks convexity and the orthogonality structure, turning the problem into a combinatorial or non-convex optimization. Conclude by noting that this makes exact solutions NP-hard and requires approximate algorithms like iterative thresholding or convex relaxations.
Pro tip: Mention that while L1-regularized PCA (Sparse PCA) is still non-convex, it can be relaxed to a convex problem (e.g., semidefinite programming) or solved via greedy methods, but these come with trade-offs in scalability and optimality. This shows awareness of practical algorithm design.
Explain that PCA finds the top eigenvectors of the covariance matrix, which is a well-understood problem solvable in polynomial time via SVD or power iteration.
Define sparsity as limiting the number of non-zero loadings in the principal components, often via L0 or L1 penalties, to improve interpretability or reduce overfitting.
Show that sparsity constraints destroy the convexity of the objective and the orthogonality of the solution, making the problem non-convex and combinatorial in nature.
State that finding the optimal sparse principal components is NP-hard, as it involves selecting a subset of variables, and exact solutions are infeasible for large dimensions.
Discuss that practitioners resort to approximate algorithms like iterative thresholding, greedy methods, or convex relaxations, which trade off optimality for tractability.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Broad question and I rambled a bit at first.
Define bias and variance, then systematically compare each model family's inherent flexibility and regularization, explaining how that affects the trade-off. Conclude with the conditions under which each minimizes total error, emphasizing the role of data size, dimensionality, and true function complexity.
Pro tip: Tie the trade-off to practical model selection: mention that in high-dimensional or noisy settings, regularization (ridge/lasso) or averaging (ensembles) often beats plain linear regression, but with abundant data and a simple true function, linear regression can be optimal.
Explain bias as error from erroneous assumptions (underfitting) and variance as sensitivity to training data (overfitting). Total error = bias^2 + variance + irreducible noise.
Linear regression has low variance but high bias if the true relationship is nonlinear. It minimizes total error when the true function is approximately linear and data is limited.
Ridge (L2) shrinks coefficients, reducing variance at the cost of slightly increased bias; lasso (L1) also performs feature selection. They minimize total error when multicollinearity or high dimensionality makes OLS unstable.
Kernel methods (e.g., SVM, kernel ridge) can model nonlinearity with high variance if the kernel is too flexible. They minimize total error when the true function is smooth and the kernel bandwidth/regularization is tuned to balance bias and variance.
Single trees have high variance; bagging (random forests) reduces variance, while boosting reduces bias. They minimize total error when the true function is complex and interactions matter, but require careful tuning to avoid overfitting.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Honestly a bit of a relief after the previous questions.
Start by framing linear regression as finding the vector in the column space of the design matrix that is closest to the response vector, which is exactly an orthogonal projection. Then explain how the residual vector is orthogonal to the column space, leading to the normal equations and the decomposition of the response into fitted values plus residuals.
Pro tip: Connect the geometry to practical implications: the residuals being orthogonal to predictors means they are uncorrelated with the fitted values, which is a key diagnostic in regression. Also mention that this geometric view extends to understanding regularization and the bias-variance tradeoff.
Define the column space of the design matrix X as a subspace of R^n, and the response vector y as a point in that space. The goal is to find the closest point in the column space to y.
Explain that the closest point is the orthogonal projection of y onto the column space, denoted as ŷ = Xβ. The residual vector e = y - ŷ is orthogonal to the column space.
Use the orthogonality condition X^T(y - Xβ) = 0 to derive the normal equations X^T X β = X^T y, and solve for β if X^T X is invertible.
Explain that the residual vector lies in the orthogonal complement of the column space, and that the total sum of squares decomposes into explained and residual sum of squares.
Discuss how this geometric view helps understand concepts like multicollinearity, regularization, and model diagnostics.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.