The naive version blows up fast if you just exponentiate raw logits.
Start by explaining the mathematical definition of entropy for a softmax distribution: H = -sum(p_i * log(p_i)), where p_i = exp(z_i) / sum(exp(z_j)). Then describe how to compute it stably using the log-sum-exp trick: compute m = max(z), then log_p_i = z_i - m - log(sum(exp(z_j - m))), and finally H = -sum(exp(log_p_i) * log_p_i).
Pro tip: Mention that you can avoid computing softmax probabilities explicitly by using the identity H = log(sum(exp(z_i))) - sum(exp(z_i) * z_i) / sum(exp(z_i)), which is also stable with the max subtraction. This shows deeper understanding and can be more efficient.
State the formulas: softmax(z)_i = exp(z_i) / sum_j exp(z_j), and entropy H(p) = -sum_i p_i log(p_i).
Explain that directly computing exp(z_i) can overflow if z_i is large, and log(p_i) can be -inf if p_i underflows to 0.
Subtract the maximum logit m = max(z) before exponentiation: compute log_sum_exp = m + log(sum(exp(z_i - m))). Then log_p_i = z_i - log_sum_exp.
Compute p_i = exp(log_p_i) and then H = -sum(p_i * log_p_i). Alternatively, use the identity H = log_sum_exp - sum(p_i * z_i) to avoid log(0).
Discuss handling of -inf logits (e.g., masked positions) and verify with simple cases like uniform logits (entropy = log(n)).
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
This is where I actually got stuck for a bit.
Explain that entropy can be computed in a streaming fashion by maintaining running sums of the logits and their exponentials, then using the log-sum-exp trick to avoid overflow. Emphasize that this requires only O(1) memory and a single pass over the logits.
Pro tip: Mention numerical stability: use the log-sum-exp trick with a running maximum to prevent overflow/underflow, and note that the same approach extends to other streaming statistics like cross-entropy.
Recall that for logits z_i, the softmax probabilities are p_i = exp(z_i) / sum_j exp(z_j), and entropy H = -sum_i p_i log p_i. Express H in terms of logits to see what needs to be accumulated.
Show that H = log(sum_j exp(z_j)) - (sum_i z_i exp(z_i)) / (sum_j exp(z_j)). Thus, we only need to maintain three running sums: S0 = sum exp(z_i), S1 = sum z_i exp(z_i), and optionally the maximum logit for stability.
Use the log-sum-exp trick: maintain a running maximum M and accumulate exp(z_i - M) and z_i exp(z_i - M). After processing all logits, adjust the sums by exp(M) to get the true values.
For each new logit z, update M = max(M, z), and update S0 and S1 by rescaling if M changes: S0 = S0 * exp(M_old - M) + exp(z - M), S1 = S1 * exp(M_old - M) + z * exp(z - M).
After all logits, compute H = log(S0) + M - S1 / S0. This gives the entropy in O(1) memory and O(n) time.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
First, clarify that entropy over logits requires a softmax normalization, which needs the global max and sum of exponentials. Then, describe a two-pass block-wise algorithm: first pass computes the global max and sum of exp(logit - max) in O(1) space, second pass computes the sum of p_i * log p_i using the stored max and sum. Finally, discuss numerical stability and how to handle streaming updates to max and sum.
Pro tip: Emphasize that the two-pass approach is necessary because softmax normalization requires global statistics; a single pass would need to store all logits or approximate. Also, mention that using the max subtraction trick prevents overflow and is standard in production systems like PyTorch's log_softmax.
Restate that we need to compute entropy H = -sum(p_i * log p_i) where p_i = exp(logit_i) / sum(exp(logit_j)). Confirm that logits are processed in fixed-size chunks and we can only use O(1) extra space, meaning we cannot store all logits or probabilities.
Recognize that softmax requires the global maximum M = max(logit_i) for numerical stability and the global sum S = sum(exp(logit_i - M)). These must be computed before any probability can be calculated.
First pass: iterate over chunks, update M and S using the streaming log-sum-exp trick. Second pass: iterate again, compute p_i = exp(logit_i - M) / S, and accumulate -p_i * log(p_i). Return the accumulated entropy.
Explain how to update M and S when a new maximum is found: S_new = S_old * exp(M_old - M_new) + sum(exp(logit - M_new)). Also discuss handling of -inf logits (probability 0) and ensuring no log(0) errors.
State that time complexity is O(N) with two passes over the data, and space is O(1). Mention that two passes require re-reading the logits, which may be costly if they are not stored; discuss alternatives like storing logits if memory allows or using a single-pass approximation.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.