My first pass just exponentiated everything directly and called it done.
Start by explaining the softmax formula and the numerical instability of directly exponentiating large logits. Then describe the max-subtraction trick: subtract the maximum logit from all logits before exponentiating, and finally normalize by the sum of exponentials. Implement the function in code, ensuring it handles edge cases like empty lists or all -inf values.
Pro tip: Mention that subtracting the max is mathematically equivalent to the original softmax and does not change the output, but it prevents overflow. Also, note that if all logits are -inf, the result should be a uniform distribution (or handle as per specification).
Define softmax as exp(x_i) / sum(exp(x_j)). Highlight that large logits cause overflow in exp, and very negative logits cause underflow to zero, leading to division by zero.
Show that softmax(x) = softmax(x - max(x)). Explain that subtracting the maximum logit ensures the largest exponent is 0, preventing overflow and making the computation stable.
Write a function that takes a list of floats, finds the maximum (handle empty list), subtracts it from each logit, computes exp for each, sums them, and divides each by the sum. Use only math.exp.
Test with large positive/negative numbers, all equal values, and empty list. Discuss that the max-subtraction adds O(n) time but is necessary for stability, and that it doesn't change the mathematical result.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.