I can derive this in my sleep so the algebra was fine.
Start by setting up the linear model and deriving the OLS estimator via minimizing the sum of squared residuals, then discuss the rank condition for identifiability, and finally state the sampling distribution under classical assumptions. Emphasize the role of assumptions in ensuring unbiasedness and efficiency.
Pro tip: Connect the rank condition to multicollinearity and its practical implications, and mention that while OLS is unbiased under Gauss-Markov, normality of errors is needed for exact t and F tests in small samples.
Define the model y = Xβ + ε, with X an n×k matrix, and state the classical assumptions (linearity, exogeneity, homoskedasticity, no autocorrelation, and optionally normality).
Minimize the sum of squared residuals (y - Xβ)'(y - Xβ) by taking the derivative with respect to β, setting it to zero, and solving the normal equations to get β_hat = (X'X)^{-1}X'y.
Explain that for (X'X) to be invertible, X must have full column rank k (no perfect multicollinearity). If not, the OLS estimator is not unique and parameters are not identifiable.
Under classical assumptions, β_hat is unbiased with covariance matrix σ²(X'X)^{-1}. If errors are normally distributed, β_hat ~ N(β, σ²(X'X)^{-1}). For large samples, asymptotic normality holds under weaker conditions.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
This is where the interview got interesting.
Start by framing the problem as high-dimensional regression where p > n, then present three distinct approaches: L1 (Lasso), L2 (Ridge), and dimensionality reduction (PCA/PLS). For each, explain the mechanism, trade-offs, and how to tune the regularization parameter via cross-validation, emphasizing that validation must account for the small sample size.
Pro tip: Mention that with p >> n, standard k-fold CV can be unstable; use repeated k-fold or leave-one-out CV, and consider that Lasso may select at most n variables, so elastic net or ridge might be preferable when predictors are correlated.
Acknowledge that p > n leads to multicollinearity, overfitting, and non-unique solutions in ordinary least squares. State that regularization or dimensionality reduction is necessary.
Describe Lasso (L1), Ridge (L2), and PCA/PLS. For each, explain how it handles p > n, its strengths (e.g., Lasso for sparsity, Ridge for correlated predictors, PCA for unsupervised reduction), and limitations.
Detail how to select the regularization parameter (lambda) using cross-validation, specifically k-fold or leave-one-out, and mention that the optimal lambda minimizes CV error. For PCA/PLS, discuss choosing the number of components via CV or scree plot.
Emphasize that validation must be done on held-out data or via nested CV to avoid optimistic bias. Mention that with small n, repeated CV or bootstrapping can provide more stable estimates.
Summarize when to use each method: Lasso for sparse true models, Ridge when many small effects, PCA/PLS for unsupervised or when interpretability of components is acceptable. Suggest starting with Ridge or Elastic Net as robust defaults.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Start by defining the Moore-Penrose pseudoinverse and its role in solving linear systems, emphasizing the minimum-norm least-squares solution. Then discuss scenarios where it is useful (e.g., underdetermined or overdetermined systems, rank-deficient matrices) and highlight practical drawbacks such as computational cost, numerical instability, and lack of regularization. Conclude with alternatives like ridge regression or iterative methods when appropriate.
Pro tip: Mention that while the pseudoinverse is theoretically elegant, in practice you often use SVD-based implementations and consider regularization (e.g., ridge) to handle ill-posed problems, especially in high-dimensional data.
Explain that the Moore-Penrose pseudoinverse provides the minimum-norm least-squares solution to Ax = b, even when A is singular or non-square.
Discuss cases like underdetermined systems (infinite solutions, pick smallest norm), overdetermined systems (least squares), and rank-deficient matrices where standard inverse fails.
Cover computational complexity (SVD is O(n^3)), numerical instability for ill-conditioned matrices, and lack of regularization leading to overfitting or meaningless solutions.
Mention that in practice, regularized methods (ridge, LASSO) or iterative solvers (conjugate gradient) are often preferred for large-scale or noisy problems.
Summarize that the pseudoinverse is useful for small, well-conditioned problems or as a theoretical baseline, but for real-world data science tasks, consider regularization and scalability.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
My gut said 'obviously it doesn't add new information' and I said exactly that.
Start by clarifying that rank deficiency is a property of the column space of the design matrix, not the number of rows. Then explain that duplicating rows does not add new independent information, so the matrix remains rank deficient, and discuss the consequences for inference such as underestimated standard errors and invalid test statistics.
Pro tip: Mention that while duplicating rows doesn't fix rank deficiency, it can artificially inflate the apparent sample size and precision, leading to overconfident but biased inferences. A mature answer will also note that the correct fix is to address the root cause, such as removing collinear features or using regularization.
Explain that rank deficiency occurs when the design matrix does not have full column rank, meaning some columns are linearly dependent. This is a structural issue, not a sample size issue.
Duplicating rows does not change the column space or the linear dependencies among columns. The rank of the matrix remains the same because the new rows are linear combinations of existing rows.
Duplicating rows artificially increases the sample size, leading to underestimated standard errors and inflated test statistics. This results in false confidence in parameter estimates and invalid hypothesis tests.
Suggest addressing the root cause: remove or combine collinear features, use regularization (e.g., ridge regression), or apply dimensionality reduction techniques like PCA.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.