TikTok·Machine Learning Engineer·Technical Phone Screen
- Write pseudocode for scaled dot-product self-attention for a single head, including tensor shapes, Q/K/V projections, scaling, softmax, and optionally an attention mask.
- Write pseudocode for multi-class cross-entropy loss over a batch, handling both class index labels and one-hot labels, returning a scalar mean loss.
- What does the position-wise feed-forward network in a Transformer block actually do, and why is it necessary given that attention already exists?
- Why do we divide attention scores by the square root of the head dimension before applying softmax?
“This is where I spent most of my mental energy.”