Start by clearly stating the linear model and its assumptions, then derive the OLS estimator by minimizing the residual sum of squares and solving the normal equations. Discuss the conditions for the closed-form solution to exist, introduce ridge regression as a remedy for multicollinearity, and explain how regularization trades off bias and variance.
Pro tip: Emphasize the geometric interpretation of OLS as an orthogonal projection onto the column space, and connect ridge regression to adding a small constant to the diagonal to stabilize the inverse. This shows deep understanding beyond memorized formulas.
Define the linear model y = Xβ + ε, and state the key assumptions: linearity, exogeneity (E[ε|X]=0), homoscedasticity, no autocorrelation, and full column rank of X.
Minimize the residual sum of squares (RSS) = ||y - Xβ||² by taking the gradient with respect to β, setting it to zero, and solving the normal equations XᵀXβ = Xᵀy to get β̂ = (XᵀX)⁻¹Xᵀy.
Explain that the closed-form solution exists if and only if XᵀX is invertible, which requires X to have full column rank (no perfect multicollinearity) and more observations than predictors (n > p).
Introduce ridge regression, which adds a penalty λ||β||² to the RSS, leading to β̂_ridge = (XᵀX + λI)⁻¹Xᵀy. This always yields an invertible matrix for λ > 0, even when XᵀX is singular.
Discuss how ridge introduces bias but reduces variance, often lowering mean squared error. As λ increases, bias increases and variance decreases; λ=0 corresponds to OLS.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Start by writing the negative log-likelihood for binary logistic regression, then derive the gradient and Hessian using the sigmoid function and its derivative. Prove convexity by showing the Hessian is positive semidefinite. Finally, compute the predicted probability for the given point, update the weights using gradient descent with learning rate 0.5, and clearly state the new weights.
Pro tip: Emphasize the connection between the Hessian and convexity: the Hessian is X^T S X where S is diagonal with entries p_i(1-p_i) ≥ 0, making it positive semidefinite. This shows both convexity and that the loss is strictly convex if the data matrix has full rank.
For binary classification with labels y ∈ {0,1}, the negative log-likelihood is L(w) = -Σ [y_i log(p_i) + (1-y_i) log(1-p_i)], where p_i = σ(w^T x_i) and σ(z) = 1/(1+e^{-z}).
The gradient is ∇L(w) = Σ (p_i - y_i) x_i = X^T (p - y). The Hessian is ∇²L(w) = Σ p_i(1-p_i) x_i x_i^T = X^T S X, where S = diag(p_i(1-p_i)).
Show that the Hessian is positive semidefinite: for any vector v, v^T ∇²L(w) v = Σ p_i(1-p_i) (v^T x_i)^2 ≥ 0 because p_i(1-p_i) ≥ 0. Thus L(w) is convex.
Given x=(1,2), y=1, w=(0.1,-0.2), compute z = w^T x = 0.1*1 + (-0.2)*2 = -0.3. Then p = σ(z) = 1/(1+e^{0.3}) ≈ 0.5744. Gradient for this sample: (p - y)x = (0.5744 - 1)*(1,2) = (-0.4256, -0.8512). Update: w_new = w - η * gradient = (0.1, -0.2) - 0.5*(-0.4256, -0.8512) = (0.3128, 0.2256).
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Went with regularization, early stopping, and data augmentation.
Start by clearly naming three distinct overfitting techniques (e.g., L2 regularization, dropout, early stopping) and for each, explain the mechanism, when it helps, and when it could hurt performance. Then, outline a cross-validation strategy for tuning lambda, emphasizing nested CV to avoid optimistic bias, and discuss practical considerations like computational cost and data size.
Pro tip: When discussing trade-offs, tie each technique to a concrete business scenario (e.g., high-dimensional sparse data vs. small datasets) to show you think beyond theory. For the CV strategy, mention that you'd use a separate validation set for early stopping if combined with other techniques, to avoid leakage.
Choose three distinct methods such as L2 regularization, dropout, and early stopping. Briefly define each and its mechanism for reducing overfitting.
For each technique, describe scenarios where it is beneficial, e.g., L2 for high-dimensional data, dropout for deep neural networks, early stopping for iterative training.
Discuss potential downsides, such as L2 biasing coefficients too much, dropout increasing training time and variance, early stopping stopping too soon and missing better optima.
Propose nested cross-validation: outer loop for performance estimation, inner loop for hyperparameter tuning. Specify k-fold (e.g., 5 or 10), stratification if needed, and how to handle computational constraints.
Mention how to scale the strategy (e.g., using random search or Bayesian optimization for lambda), and how to combine with other techniques without leakage.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Structure your answer in two clear parts: first, explain the percentile bootstrap interval for the mean, demonstrate with two resamples from the given sample, and justify its non-parametric nature. Second, describe boosting's core idea and manually compute one AdaBoost iteration with the given misclassification, showing all steps and calculations.
Pro tip: When explaining the percentile bootstrap, emphasize that it directly estimates the sampling distribution of the mean without assuming normality, which is crucial for small or skewed samples. For AdaBoost, double-check your arithmetic and clearly state each formula before plugging in numbers to avoid errors under pressure.
Describe the percentile method: resample with replacement many times, compute the mean for each resample, and take the 2.5th and 97.5th percentiles as the interval. Mention that this approximates the sampling distribution of the mean.
Generate two resamples from [2,3,5,7,11] with replacement, compute their means, and show them as examples. For instance, Resample 1: [2,2,7,11,3] mean=5.0; Resample 2: [5,5,3,7,11] mean=6.2.
Highlight that the bootstrap relies on the empirical distribution of the data, not on any assumed parametric form. By resampling, we mimic the process of drawing from the population, so the distribution of resample means approximates the true sampling distribution.
Explain that boosting sequentially combines weak learners, each focusing on the errors of the previous ones, to create a strong learner. Mention that it reduces bias and can achieve high accuracy.
With three equally weighted points and one misclassification, compute epsilon = 1/3, alpha = 0.5 * ln((1-epsilon)/epsilon) = 0.5 * ln(2) ≈ 0.3466. Update weights: misclassified point weight becomes (1/3)*exp(alpha) ≈ 0.473, others become (1/3)*exp(-alpha) ≈ 0.236. Sum = 0.945, renormalize to get distribution: [0.25, 0.50, 0.25].
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Felt like the coolest question of the set because it's actually practical.
Start by defining each ensemble method briefly, then compare them across bias, variance, and noise sensitivity in a structured table-like format. Finally, provide a concrete scenario for each, ideally tied to real-world data science problems like fraud detection or customer churn. Emphasize the trade-offs and practical considerations for model selection.
Pro tip: Mention that random forests are often a strong baseline due to their robustness and minimal tuning, but boosting can outperform when carefully tuned and when the data is clean. Also, highlight that Amazon values scalability and production readiness, so discuss computational cost and ease of deployment.
Briefly explain bagging, boosting, and random forests: bagging trains models in parallel on bootstrapped samples; boosting trains sequentially, focusing on errors; random forests are bagging with decision trees and feature randomness.
Bagging and random forests primarily reduce variance while keeping bias similar; boosting reduces bias but can increase variance if not regularized. Random forests further reduce variance due to feature randomness.
Bagging and random forests are robust to noisy labels because they average over many models; boosting is sensitive because it focuses on misclassified points, which may be noise, leading to overfitting.
For bagging: high-variance models like unpruned decision trees on noisy data. For boosting: clean data where high accuracy is needed, e.g., click-through rate prediction. For random forests: general-purpose baseline with mixed data types and some noise.
Conclude with when to choose each: bagging for variance reduction, boosting for bias reduction with clean data, random forests for robustness and ease of use. Mention computational cost and tuning effort.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.