Explain that entropy can be computed online by maintaining running aggregates of the logits, such as the maximum logit and the sum of exponentials, using numerically stable updates. Then, at any point, the entropy can be derived from these aggregates without storing all logits.
Pro tip: Mention that for streaming entropy, you can use the log-sum-exp trick with a running maximum to avoid overflow, and that the same approach extends to other softmax-based quantities like cross-entropy.
Recall that entropy H = log(sum(exp(logits))) - sum(softmax(logits) * logits). This can be computed from the log-sum-exp and the expected logit under the softmax.
The entropy depends only on the log-sum-exp of the logits and the sum of logits weighted by softmax probabilities. These can be updated incrementally.
Keep a running maximum logit M and a running sum S = sum(exp(logit - M)). Also maintain a running weighted sum W = sum(logit * exp(logit - M)). Update these as new logits arrive.
When a new maximum arrives, rescale S and W by exp(old_M - new_M) to avoid overflow. This ensures stable updates.
At any point, compute log-sum-exp as M + log(S), and the expected logit as W/S. Then entropy = log-sum-exp - expected logit.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.