This one threw me a bit because I expected LeetCode-style graph or DP stuff and instead got asked to basically implement a core transformer building block.
Start by clarifying the input shapes and the desired output shape, then derive the scaled dot-product attention formula step by step. Implement the linear projections for Q, K, V, split into heads, compute attention per head, concatenate, and apply the final output projection. Test with small random matrices to verify correctness.
Pro tip: Mention that you would use stable softmax (subtract max) and that you would vectorize the computation across heads for efficiency. Also, note that in practice you would use optimized libraries like PyTorch's scaled_dot_product_attention, but implementing from scratch demonstrates understanding.
Confirm the shapes of query, key, value matrices and the number of heads. Determine the expected output shape and whether any masking is required.
Apply linear projections to Q, K, V using weight matrices (if provided) or assume they are already projected. Reshape and transpose to split the last dimension into (num_heads, head_dim).
For each head, compute attention scores = Q @ K^T / sqrt(head_dim), apply softmax (with numerical stability), then multiply by V to get the head output.
Concatenate the outputs from all heads along the head dimension, then apply a final linear projection to produce the final output.
Test with small random inputs and verify shapes and numerical correctness. Compare against a reference implementation if possible.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.