Start by outlining the standard Transformer architecture and the typical bug categories mentioned. Then systematically trace through the code, checking each component (attention, feed-forward, etc.) for shape mismatches, masking errors, scaling issues, softmax axis, initialization, and typos. For each bug, explain the fix and why it matters.
Pro tip: Demonstrate a methodical debugging process: use print statements or shape assertions to verify tensor dimensions at each step, and explain how each bug would affect training or inference. This shows you can not only find bugs but also understand their impact.
Recall the standard Transformer components: multi-head attention, positional encoding, layer normalization, feed-forward networks, and residual connections. Know the expected tensor shapes and operations.
Verify scaling of QK^T by sqrt(d_k), correct softmax axis (last dimension), and proper masking (e.g., causal mask for decoder). Ensure shapes align for matrix multiplications.
Check dimensions in feed-forward layers (expansion and contraction), and verify layer normalization is applied correctly (e.g., over the last dimension). Look for missing residual connections.
Ensure weights are initialized appropriately (e.g., Xavier/Glorot). Scan for variable name typos that could cause incorrect tensor usage or silent errors.
Mentally or actually run a small input through the model, checking shapes and outputs at each step. Confirm that fixes resolve the issues without introducing new ones.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.