Begin by writing the sigmoid function and the Bernoulli log-likelihood for binary logistic regression. Then derive the gradient and Hessian of the L2-regularized negative log-likelihood with respect to the coefficients, and finally explain why the objective is convex by showing the Hessian is positive semidefinite.
Pro tip: Emphasize that the L2 regularization term adds a positive definite matrix to the Hessian, which guarantees strict convexity and a unique solution. Also, connect the derivation to practical optimization benefits like convergence guarantees.
Write the sigmoid function σ(z) = 1/(1+e^{-z}) and the Bernoulli log-likelihood for binary logistic regression: ℓ(β) = Σ [y_i log σ(x_i^T β) + (1-y_i) log(1-σ(x_i^T β))].
Define the L2-regularized negative log-likelihood: J(β) = -ℓ(β) + (λ/2) ||β||^2, where λ > 0 is the regularization strength.
Compute the gradient of J(β) with respect to β: ∇J(β) = Σ (σ(x_i^T β) - y_i) x_i + λ β.
Compute the Hessian: ∇²J(β) = Σ σ(x_i^T β)(1-σ(x_i^T β)) x_i x_i^T + λ I.
Show that the Hessian is positive semidefinite because each term σ(1-σ) x x^T is PSD and λI is positive definite, making J(β) convex (strictly convex if λ > 0).
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Straightforward computation, z = -1.2 + 0.8*2 = 0.4, then sigmoid gives roughly 0.599.
First, identify that this is a logistic regression problem and recall the formulas: log-odds = intercept + coefficient * x, probability = 1 / (1 + exp(-log-odds)), odds = exp(log-odds), and odds ratio = exp(coefficient). Then plug in the given values, compute step-by-step, and clearly state each result with proper interpretation.
Pro tip: Always interpret the results in the context of the problem—e.g., 'For x=2.0, the predicted probability is about 0.31, meaning a 31% chance of the positive outcome.' This shows you understand the business implication, not just the math.
Recognize that this is a logistic regression with intercept -1.2 and coefficient 0.8. Write down the key formulas: log-odds = β0 + β1*x, probability = 1/(1+e^{-log-odds}), odds = e^{log-odds}, odds ratio = e^{β1}.
Calculate log-odds = -1.2 + 0.8 * 2.0 = -1.2 + 1.6 = 0.4.
Use the logistic function: probability = 1 / (1 + e^{-0.4}) ≈ 1 / (1 + 0.6703) ≈ 0.5987. So the predicted probability is about 0.599 or 59.9%.
Odds = e^{0.4} ≈ 1.4918. Odds ratio for a one-unit increase in x is e^{0.8} ≈ 2.2255.
State that for x=2.0, the model predicts a 59.9% probability of the positive outcome, with odds of about 1.49. A one-unit increase in x multiplies the odds by about 2.23, indicating a positive relationship.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
The threshold math is cost-weighted: threshold = 1 / (1 + cost_ratio * base_rate_ratio), roughly comes out around 0.36 or so depending on how you set it up.
Start by deriving the Bayes-optimal threshold using the given costs and class prior, then explain how to calibrate model probabilities to make that threshold meaningful. Finally, compare Platt scaling and isotonic regression in terms of flexibility, data requirements, and overfitting risk, and discuss how to choose between them in practice.
Pro tip: Emphasize that the optimal threshold depends on calibrated probabilities and that in production you should monitor calibration drift and re-evaluate the threshold as costs or class balance change.
Use the formula threshold = cost_FP / (cost_FP + cost_FN) adjusted for class prior, or equivalently compare expected costs. With cost_FN = 10 * cost_FP and prior positive = 0.02, compute the threshold.
Describe why raw model scores (e.g., from SVM or tree ensembles) may not be probabilities and how calibration maps them to well-calibrated probabilities, enabling cost-sensitive thresholding.
Discuss Platt scaling (parametric, sigmoid-based, works well with limited data, assumes monotonic sigmoid shape) versus isotonic regression (non-parametric, more flexible, needs more data, can overfit).
Highlight that Platt scaling is less prone to overfitting and works with small datasets, while isotonic regression can capture complex calibration curves but requires more data and may overfit. Mention that both require a separate calibration set.
Recommend choosing based on data size and model complexity, and stress the importance of monitoring calibration and threshold in production as data distributions shift.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Start by defining sigmoid(z) and the naive expressions, then explain why they fail for large |z| due to floating-point overflow/underflow. Present the stable forms using softplus and log1p, and emphasize their importance in numerical optimization and model training.
Pro tip: Mention that these stable forms are used in libraries like TensorFlow and PyTorch, and that they prevent NaN losses and improve gradient stability, which is crucial for large-scale models at Snapchat.
State that sigmoid(z) = 1/(1+exp(-z)), and the naive forms are log(sigmoid(z)) and log(1 - sigmoid(z)).
For large positive z, exp(-z) underflows to 0, making log(sigmoid(z)) = log(1) = 0, but actually it should be close to -exp(-z). For large negative z, exp(-z) overflows to inf, making sigmoid(z) = 0, so log(sigmoid(z)) = -inf, and log(1 - sigmoid(z)) = log(1) = 0, but should be close to z.
Show that log(sigmoid(z)) = -log(1 + exp(-z)) = -softplus(-z), and log(1 - sigmoid(z)) = -z - log(1 + exp(-z)) = -softplus(z). Use log1p for small arguments.
Explain that softplus is computed stably as max(z,0) + log1p(exp(-|z|)), and that these forms avoid overflow/underflow.
Relate to logistic regression, neural networks, and loss functions like binary cross-entropy, where stable computation prevents NaN and improves training.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
I talked about how the MLE gets pulled toward the majority class, coefficients get underestimated for the minority class, and the intercept is particularly biased.
Start by explaining the analytical impact of class imbalance on MLE in logistic regression, focusing on how it biases intercept and inflates coefficient variance. Then discuss reweighting strategies (class weights, oversampling) and regularization (L1/L2, elastic net) with their trade-offs, backing each with mathematical reasoning. Finally, tie it to evaluation metrics and practical considerations for Snapchat's scale.
Pro tip: Emphasize that reweighting changes the effective prior and can distort probability calibration, so you must recalibrate if probabilities are used downstream. Also, mention that with massive data, simple reweighting often suffices, but with small data, regularization combined with reweighting is more robust.
Derive how class imbalance affects the log-likelihood: the intercept shifts to match the marginal probability, and rare class coefficients have high variance due to few effective samples. Mention that MLE is consistent but inefficient for rare events.
Explain class weights (inverse frequency) and their effect on the loss function: they scale the gradient contributions, effectively changing the prior. Discuss oversampling/undersampling and SMOTE, noting that oversampling can lead to overfitting and SMOTE may not be suitable for high-dimensional sparse data.
Describe L1 (lasso) for sparsity, L2 (ridge) for shrinkage, and elastic net for correlated features. Explain how regularization adds a penalty term to the log-likelihood, reducing variance at the cost of bias, which is crucial when rare class has few samples.
Discuss combining reweighting with regularization, e.g., weighted logistic regression with L2 penalty. Mention hyperparameter tuning via cross-validation with stratified folds and using metrics like AUC-PR, F1, or recall at fixed precision.
Highlight that reweighting distorts probability estimates, so calibrate using Platt scaling or isotonic regression if probabilities are needed. Evaluate with appropriate metrics and consider business impact (e.g., cost-sensitive learning).
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.