I defaulted to writing the minimization form (negative log-likelihood) without being asked, which is fine but I fumbled the transition when they asked me to show the maximization version too.
Start by clearly stating the setup: binary labels y ∈ {0,1}, features x, parameters w (and bias b). Then write the log-likelihood (to be maximized) and the negative log-likelihood (to be minimized), and show how L2 regularization adds a penalty term to the negative log-likelihood. Be explicit about the sign and the optimization direction.
Pro tip: Mention that in practice, optimization libraries (e.g., scikit-learn, TensorFlow) minimize the negative log-likelihood plus regularization, and that the regularization parameter λ (or C = 1/λ) controls the strength of the penalty. This shows awareness of implementation details.
State that for binary logistic regression, P(y=1|x) = σ(w·x + b), where σ(z) = 1/(1+e^{-z}). Assume a dataset of N independent samples {(x_i, y_i)}.
The likelihood is L(w) = ∏_{i=1}^N σ(w·x_i + b)^{y_i} (1-σ(w·x_i + b))^{1-y_i}. The log-likelihood is ℓ(w) = ∑_{i=1}^N [y_i log σ(w·x_i + b) + (1-y_i) log(1-σ(w·x_i + b))].
The maximum-likelihood objective is to maximize ℓ(w). Equivalently, minimize the negative log-likelihood: J(w) = -ℓ(w).
The L2-regularized objective is to maximize ℓ(w) - (λ/2) ||w||^2, or equivalently minimize J_reg(w) = -ℓ(w) + (λ/2) ||w||^2, where λ ≥ 0 controls the penalty strength.
Emphasize that maximizing the log-likelihood is equivalent to minimizing the negative log-likelihood, and that regularization is typically added to the negative log-likelihood (so it penalizes large weights). Note that some libraries use C = 1/λ.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Start by defining the binary logistic regression model and its probabilistic output, then derive the cross-entropy loss from maximum likelihood estimation. Present the final loss function clearly, explaining each term and its role in penalizing incorrect predictions.
Pro tip: Emphasize that cross-entropy loss is equivalent to the negative log-likelihood of the Bernoulli distribution, and mention that it's convex, which ensures reliable optimization. This shows depth beyond just memorizing the formula.
State that logistic regression models the probability of the positive class as p(y=1|x) = σ(w·x + b), where σ is the sigmoid function.
For a dataset of N independent samples, the likelihood is the product of p(y_i=1|x_i) for positive examples and (1 - p(y_i=1|x_i)) for negative examples.
The cross-entropy loss is the negative log-likelihood: L(w,b) = -Σ [y_i log(p_i) + (1-y_i) log(1-p_i)], where p_i = σ(w·x_i + b).
Clarify that y_i is the true label (0 or 1), p_i is the predicted probability, and the loss penalizes deviations from the true labels, with log terms ensuring large penalties for confident wrong predictions.
Note that this loss is convex and typically minimized using gradient descent, with gradients having a simple form: ∇_w L = Σ (p_i - y_i) x_i.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Start by clearly defining the setup: a single-layer model with softmax output and cross-entropy loss. Then derive the gradient of the loss with respect to the logits, which simplifies to (predicted probability - true label). Finally, use the chain rule to express the gradients with respect to weights and bias in terms of the input features and this logit gradient.
Pro tip: Emphasize that the gradient with respect to the bias is simply the sum of the logit gradients over the batch, which is a useful sanity check and often overlooked. Also, mention that this derivation is the foundation for backpropagation in neural networks.
Clearly state the model: logits z = Wx + b, softmax probabilities p = softmax(z), and cross-entropy loss L = -sum(y_i log(p_i)). Define all variables and dimensions.
Derive dL/dz = p - y, where p is the predicted probability vector and y is the one-hot true label vector. Explain the steps using the chain rule and properties of softmax.
Use z = Wx + b to get dL/dW = (dL/dz) x^T and dL/db = dL/dz. Show that for a batch, these become matrix multiplications and sums.
If applicable, mention how the gradients change when averaging over a batch or adding L2 regularization. This shows awareness of practical implementation.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Honestly the question I was least prepared for even though it sounds easy.
Start by defining each term clearly and in relation to the others: loss is the per-example or aggregate error, objective is the function being optimized (often loss plus regularization), and regularized objective adds a penalty term to the objective. Then explain how they connect in logistic regression, emphasizing that the regularized objective is what is actually minimized during training, and discuss the role of regularization in preventing overfitting.
Pro tip: Mention that while the loss is typically the negative log-likelihood, the objective might include additional terms like regularization, and that the choice of regularization affects the bias-variance trade-off. Also, note that in practice, the terms are sometimes used interchangeably, so clarify the context.
Explain that loss (or cost) function measures the error between predictions and true labels for a single example or the entire dataset. In logistic regression, it's typically the negative log-likelihood (log loss).
Describe the objective function as the overall function that the optimization algorithm aims to minimize or maximize. It often includes the loss but may also include other terms. In logistic regression without regularization, the objective is the loss.
Introduce regularization as an additional penalty term added to the objective to control model complexity. The regularized objective is the sum of the loss and a regularization term (e.g., L1 or L2 penalty).
Clarify that the loss is a component of the objective, and the regularized objective is a specific type of objective that includes regularization. Emphasize that in practice, the regularized objective is what is minimized during training.
Explain why regularization is used: to prevent overfitting, improve generalization, and handle multicollinearity. Mention that the choice of regularization strength (hyperparameter) is crucial and often tuned via cross-validation.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.