The numerically stable part is where people slip up.
Start by explaining the numerical instability of naive softmax and the standard log-sum-exp trick. Then derive the gradient of the loss with respect to logits, showing that it simplifies to (softmax - one_hot) / N. Finally, implement both forward and backward passes in NumPy, ensuring stability by subtracting the max logit before exponentiation.
Pro tip: Emphasize that the gradient simplifies to (softmax - one_hot) / N, which is both elegant and efficient. Also, mention that you can reuse the softmax probabilities from the forward pass in the backward pass to save computation.
Describe why computing exp(logits) directly can overflow, and how subtracting the maximum logit before exponentiation stabilizes the computation without changing the result.
Show that the loss for a single example is -log(softmax(logits)[true_class]), and using the log-sum-exp trick, it becomes -logits[true_class] + max_logit + log(sum(exp(logits - max_logit))).
Compute the gradient by differentiating the loss, yielding (softmax(logits) - one_hot(true_class)) / N for a batch of N examples.
Write vectorized code for a batch: compute max_logits, shifted_logits, exp_shifted, sum_exp, log_probs, and loss; then compute softmax probabilities and the gradient using the simplified formula.
Verify correctness by comparing with a naive implementation on small inputs, and check numerical stability with large logits. Optionally, use finite differences to validate the gradient.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.