Start by defining the general objective function for linear regression (squared error) and logistic regression (log-loss), then systematically add each penalty term in both constrained and penalized forms. For each penalty, describe the expected coefficient patterns (e.g., sparsity, shrinkage, grouping) and discuss optimization challenges (convexity, computational complexity, non-differentiability).
Pro tip: Emphasize that L0 is non-convex and NP-hard, L1 induces sparsity and is convex but non-smooth (use proximal methods), L2 is smooth and convex (closed-form for linear), and L∞ promotes grouping and is convex but non-smooth. Also note that in practice, L1 and L2 are most common, and elastic net combines them.
Write the unregularized objective for linear regression (mean squared error) and logistic regression (negative log-likelihood).
For each penalty (L0, L1, L2, L∞), add λ times the penalty term to the base objective, resulting in the penalized form.
For each penalty, express the equivalent constrained optimization problem where the penalty is replaced by a constraint on the norm (e.g., ||β||_p ≤ t).
Describe how each penalty affects the coefficients: L0 gives sparsity (subset selection), L1 gives sparsity and shrinkage, L2 gives shrinkage but no sparsity, L∞ gives grouping and shrinkage.
For each penalty, comment on convexity, smoothness, and computational complexity: L0 is non-convex and NP-hard; L1 is convex but non-smooth (use subgradient or proximal methods); L2 is convex and smooth (closed-form for linear); L∞ is convex but non-smooth (can be reformulated as linear constraints).
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Start by framing the problem: collinearity among predictors and outliers in both features and target. Then discuss why plain L1 or L2 may be insufficient, and propose elastic net as a balanced solution. Explain how to tune the mixing parameter (l1_ratio) based on the relative importance of sparsity vs. grouping and robustness to outliers.
Pro tip: Emphasize that elastic net's L1 component handles collinearity by selecting one variable from a group, while the L2 component stabilizes the solution; also mention that outliers may require robust preprocessing or a robust loss function, as regularization alone doesn't address outliers.
Acknowledge that collinearity makes coefficient estimates unstable and outliers can distort the loss function, leading to poor generalization.
Compare L1 (lasso) for sparsity, L2 (ridge) for grouping, and elastic net as a hybrid that can handle both collinearity and feature selection.
Explain that elastic net is preferable because it combines L1 and L2 penalties, providing both variable selection and coefficient shrinkage, which is beneficial with collinear groups.
Discuss how to set the l1_ratio (e.g., via cross-validation) to balance sparsity and grouping; consider domain knowledge and the need for interpretability.
Mention that regularization alone doesn't solve outliers; consider robust scaling, outlier detection, or using a robust loss function in conjunction with elastic net.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Felt straightforward but I second-guessed myself on the conditional distribution wording.
Start by defining both models with their formulas, link functions, and assumed conditional distributions. Then contrast their outputs and loss functions, and finally explain why linear regression is problematic for classification, covering both practical and theoretical issues.
Pro tip: Mention that logistic regression can be derived from a generalized linear model with a Bernoulli distribution and logit link, and that using linear regression for classification violates key assumptions and can produce nonsensical probabilities.
State the model formula: y = Xβ + ε, where ε ~ N(0, σ²). The link function is identity, and the conditional distribution of y given X is Normal.
State the model formula: log(p/(1-p)) = Xβ, where p = P(y=1|X). The link function is logit, and the conditional distribution is Bernoulli.
Linear regression predicts continuous values and minimizes squared error; logistic regression predicts probabilities and minimizes log-loss (cross-entropy).
Discuss issues: predictions outside [0,1], violation of normality and homoscedasticity assumptions, sensitivity to outliers, and inappropriate loss function leading to poor probability estimates.
Conclude that linear regression is for continuous outcomes, while logistic regression is for binary classification, and mention extensions like multinomial logistic regression.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Start by explaining that L1 and L2 regularization add a penalty proportional to the magnitude of coefficients, so the scale of features directly affects the penalty. Then discuss how skipping scaling causes the regularization to penalize features unevenly, leading to suboptimal models. Finally, emphasize that scaling ensures fair regularization and better convergence.
Pro tip: Mention that while tree-based models are invariant to feature scaling, linear models with regularization are not, and at Amazon, where both are used, it's crucial to know when scaling matters. Also, note that scaling should be fit on training data only to avoid data leakage.
Briefly define feature scaling (e.g., standardization, normalization) and L1/L2 regularization (Lasso/Ridge) as penalties on coefficient magnitudes.
Describe how the penalty term in the loss function is applied to coefficients, which are estimated on the scaled features. If features are on different scales, the coefficients will be on different scales, so the penalty affects them unequally.
Detail what goes wrong: features with larger scales get smaller coefficients (to compensate), so they are penalized less, while features with smaller scales get larger coefficients and are penalized more. This biases the model and can lead to poor performance.
Mention that unscaled features can cause slow convergence in gradient-based optimization, especially with regularization, as the loss surface becomes elongated.
Conclude with best practices: always scale features before applying regularization, use pipelines to avoid data leakage, and note that tree-based models are an exception.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.