The core of this is not complicated if you've read the transformer paper, but doing it live under pressure with no framework to lean on is a different story.
Start by clarifying the input shapes and the mathematical formulation of scaled dot-product attention, then implement each step vectorized in NumPy: linear projections, reshaping into heads, computing attention scores with scaling and softmax, applying the attention weights to values, and finally concatenating heads and projecting the output. Emphasize numerical stability and shape consistency throughout, and test with small random inputs to verify correctness.
Pro tip: Mention that you would use a numerically stable softmax by subtracting the max before exponentiation, and that you would verify the implementation against a reference (e.g., PyTorch) or by checking that attention weights sum to 1 across the key dimension.
Confirm the dimensions of the input token representations (batch_size, seq_len, d_model) and the projection weight matrices (d_model, d_model) for Q, K, V, and output. Also clarify the number of heads and head dimension.
Perform linear projections: Q = X @ W_q, K = X @ W_k, V = X @ W_v. Then reshape and transpose to split the model dimension into multiple heads, resulting in shapes (batch_size, num_heads, seq_len, d_k).
Calculate attention scores as Q @ K^T / sqrt(d_k), apply softmax over the last dimension (keys) with numerical stability, and then compute the weighted sum of values: attention_weights @ V.
Transpose and reshape the per-head outputs back to (batch_size, seq_len, d_model), then apply the output projection: output = concatenated_heads @ W_o.
Test with small random inputs, check shapes and that attention weights sum to 1. Discuss computational complexity (O(n^2 d)) and memory considerations, and mention potential optimizations like using einsum or avoiding unnecessary copies.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.