Start by clarifying the input shapes and the overall architecture, then implement the linear projections and reshaping for multi-head attention. Walk through the scaled dot-product attention with masking, and finally merge the heads and apply the output projection. Emphasize the computational complexity and practical considerations.
Pro tip: Mention that you would use `torch.einsum` or `torch.matmul` with proper broadcasting for efficiency, and that you would verify the implementation against a reference like `nn.MultiheadAttention` in a unit test.
Define input tensor shapes (batch_size, seq_len, d_model) and the number of heads. Explain that d_model must be divisible by num_heads, and each head has dimension d_k = d_model // num_heads.
Create learnable weight matrices W_q, W_k, W_v of shape (d_model, d_model) and biases. Project inputs to Q, K, V, then reshape to (batch_size, num_heads, seq_len, d_k) by splitting the last dimension.
Compute attention scores as Q @ K^T / sqrt(d_k). Apply causal mask (if needed) by setting future positions to -inf before softmax, and padding mask by setting padded positions to -inf. Then compute softmax and weighted sum with V.
Transpose and reshape the attention output back to (batch_size, seq_len, d_model), then apply a final linear projection W_o to combine information from all heads.
Analyze time and space complexity: O(n^2 * d) for attention, and O(n^2) memory for the attention matrix. Mention that multi-head attention increases representational power at the cost of more parameters and computation.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.