I knew the answer existed but fumbled the variance argument at first.
Start by explaining the mathematical intuition: without scaling, the dot products grow with dimension, pushing softmax into saturated regions with tiny gradients. Then describe how dividing by sqrt(d_k) stabilizes variance and preserves useful gradient flow, and finally connect this to training stability and model performance.
Pro tip: Mention that the scaling factor is derived from the variance of the dot product assuming independent components with zero mean and unit variance, and that it ensures the softmax remains in a regime where gradients are well-behaved.
Explain that Q and K are vectors of dimension d_k, and their dot product is the sum of d_k products of random variables. Without scaling, the variance of the dot product grows linearly with d_k.
Describe how large magnitude inputs to softmax cause the output to become nearly one-hot, leading to vanishing gradients for most entries and slow learning.
Show that dividing by sqrt(d_k) normalizes the variance of the dot product to approximately 1, keeping the softmax inputs in a reasonable range.
Explain that this scaling prevents saturation, maintains healthy gradients, and allows the model to learn effectively, especially with large d_k.
Conclude that the scaling is a simple but crucial trick that improves optimization and stability in Transformer models.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.