Meta·Machine Learning Engineer·Technical Phone Screen
- Derive the scaled dot-product attention formula and explain why we divide by the square root of the key dimension.
- Walk through every component inside a single transformer block, and explain the difference between pre-norm and post-norm placement.
- Compare causal and bidirectional attention masking. Where is each actually used?
- Compare different positional encoding schemes, including sinusoidal, learned absolute, relative bias approaches, and rotary encodings. What are the tradeoffs around extrapolating to longer sequences?
- How does dropout behave differently during training versus inference, and why do we scale activations the way we do?
“I could write out softmax(QK^T / sqrt(d_k))V fine but stumbled explaining the scaling part.”