This was a lot to hold in your head at once.
Start by outlining a systematic debugging process: understand the expected behavior of each component, then inspect the code for common pitfalls in attention, masking, shapes, layer norm, and positional encoding. For each bug, explain the correct implementation, propose a minimal fix, and describe a sanity check to verify the fix.
Pro tip: Demonstrate a deep understanding of Transformer internals by explaining not just what is wrong but why it matters for model performance and training stability. Use unit tests or small examples to validate fixes, showing a rigorous engineering approach.
Check scaled dot-product attention: ensure queries, keys, values are correctly projected and scaled by sqrt(d_k). Verify softmax is applied to the correct dimension and that attention weights sum to 1.
Examine padding and causal masks: ensure they are correctly shaped and applied before softmax (e.g., using -inf for masked positions). Verify that masks are not applied to future tokens in decoder self-attention.
Trace tensor shapes through the network: check that batch, sequence, and feature dimensions are consistent, especially after multi-head splitting and concatenation. Ensure residual connections and layer norm inputs match.
Verify layer norm is applied correctly (pre-norm vs post-norm) and that it normalizes over the feature dimension. Ensure it is applied after residual addition if using post-norm.
Confirm positional encodings are added to input embeddings and are correctly computed (e.g., sinusoidal or learned). Check that they are not accidentally omitted or added after the first layer.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.