Binary cross-entropy, and the 'why' is the part people fumble.
Start by directly naming the loss function: binary cross-entropy (log loss). Then explain the reasoning from two angles: probabilistic (maximum likelihood estimation under a Bernoulli model) and practical (convexity, gradient properties). Finally, contrast it with alternatives like MSE to highlight why cross-entropy is preferred.
Pro tip: Mention that cross-entropy is not just a choice but a natural consequence of assuming a Bernoulli distribution for the target and using maximum likelihood estimation. This shows deep understanding beyond memorization.
State clearly that logistic regression uses binary cross-entropy (log loss) for binary classification, and categorical cross-entropy for multi-class (softmax).
Describe how the loss arises from maximum likelihood estimation: assuming the target follows a Bernoulli distribution, the negative log-likelihood yields the cross-entropy loss.
Highlight that cross-entropy is convex (for logistic regression), ensuring a unique global minimum, and its gradient has a simple form (difference between predicted probability and true label), which aids efficient optimization.
Contrast with mean squared error (MSE): MSE is non-convex for logistic regression and suffers from vanishing gradients when predictions are saturated, making it less suitable.
Conclude that cross-entropy is the standard because it aligns with the probabilistic nature of the model, provides well-behaved gradients, and leads to better calibration and performance.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Start by directly stating that linear regression uses Mean Squared Error (MSE) as its loss function. Then explain the mathematical and statistical justifications: MSE is the negative log-likelihood under Gaussian noise, it's convex and differentiable, and it has a closed-form solution. Finally, connect this to practical implications like sensitivity to outliers and the bias-variance trade-off.
Pro tip: Mention that while MSE is standard, it assumes Gaussian noise; if the data has outliers or heavy-tailed noise, alternatives like MAE or Huber loss might be more appropriate. This shows you understand the assumptions and can adapt to real-world scenarios.
Clearly identify Mean Squared Error (MSE) as the standard loss function for linear regression, and optionally mention its formula: (1/n) * Σ(y_i - ŷ_i)^2.
Describe how MSE arises from the assumption that errors are independent and normally distributed with constant variance, making it equivalent to maximizing the likelihood.
Highlight that MSE is convex, differentiable, and leads to a closed-form solution (normal equations), which makes optimization efficient and guarantees a global minimum.
Acknowledge that MSE is sensitive to outliers and may not be suitable for all data; briefly mention alternatives like MAE or Huber loss and when they might be preferred.
Relate the choice of loss function to model performance, interpretability, and the bias-variance trade-off, showing awareness of real-world trade-offs.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Structure your answer by first defining MSE and MAE mathematically, then contrast their sensitivity to outliers and gradient properties, and finally discuss practical selection criteria based on data characteristics and business objectives. Use concrete examples to illustrate when each metric is preferable.
Pro tip: Mention that while MSE is differentiable everywhere and easier to optimize with gradient descent, MAE's constant gradient can be advantageous in robust regression but may require subgradient methods. Also, highlight that the choice often depends on whether outliers are noise or signal.
State the formulas: MSE = (1/n) Σ (y_i - ŷ_i)^2 and MAE = (1/n) Σ |y_i - ŷ_i|. Emphasize that MSE squares errors, while MAE takes absolute values.
Explain that MSE penalizes large errors more heavily due to squaring, making it sensitive to outliers. MAE treats all errors linearly, so it is more robust to outliers.
Describe that MSE has a gradient proportional to the error, which shrinks as error decreases, leading to smooth convergence. MAE has a constant gradient magnitude (except at zero), which can cause oscillations near the minimum but is robust to outliers.
Discuss that MSE is preferred when large errors are particularly undesirable and outliers are meaningful, while MAE is better when outliers are noise or when a robust model is needed. Also consider optimization ease and interpretability.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.