Start by tracing the tensor shapes through the MLP block, comparing the broken and correct implementations to pinpoint the mismatch. Then explain how the shape error causes a runtime failure or silent incorrectness during training, and propose a fix with justification.
Pro tip: Demonstrate that you understand the MLP's role in Transformers and the importance of shape consistency; mention that such bugs often manifest as cryptic errors, so systematic shape checking is key.
Recall that a Transformer MLP typically consists of two linear layers with a nonlinearity in between, often expanding then projecting back to the model dimension.
Manually compute the expected shape after each operation (input, first linear, activation, second linear) and compare with the broken code to find the mismatch.
Determine whether the shape error causes an immediate runtime error (e.g., matrix multiplication dimension mismatch) or a silent issue (e.g., broadcasting leading to wrong gradients).
Adjust the offending layer's dimensions or reshape operation to restore shape consistency, and explain why this resolves the training failure.
Describe how the bug would prevent the model from learning, either by crashing the training loop or by producing incorrect outputs that hinder optimization.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
First, explain how attention masks work and the three common bugs: wrong shape, wrong sign, and wrong tensor. Then, describe how each bug manifests during training, such as loss not decreasing, NaN gradients, or attention weights not summing to 1. Finally, outline a systematic debugging process to identify and fix the issue.
Pro tip: Emphasize that you always validate mask shapes and values with assertions or unit tests before training, and monitor attention entropy and gradient norms to catch mask issues early.
Explain that attention masks prevent the model from attending to padding tokens or future tokens in autoregressive settings. They are typically added to attention logits before softmax, with -inf for masked positions.
Wrong shape: mask not broadcastable to attention scores (e.g., missing head dimension). Wrong sign: using +inf or large positive values instead of -inf, causing softmax to focus on masked positions. Wrong tensor: applying mask to values instead of logits, or to the wrong attention layer.
Wrong shape: runtime error or silent broadcasting that masks unintended positions. Wrong sign: loss diverges or model attends to padding/future tokens, leading to poor generalization. Wrong tensor: gradients vanish or explode, loss plateaus, or model fails to learn dependencies.
Check mask shape against attention scores, verify mask values are 0 and -inf, and ensure mask is added to logits before softmax. Use small synthetic examples to validate attention outputs.
Correct the mask shape, sign, or application point. Add assertions and unit tests to catch future issues. Monitor attention entropy and gradient norms during training to confirm the fix.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
The .item() one is a classic trap and I said so out loud, which maybe came off as overconfident.
Systematically inspect the code for common autograd pitfalls: detached tensors, premature .item() calls, incorrect reduction dimensions, and in-place operations. Use a minimal reproduction and compare gradients to a reference implementation to isolate the bug. Then fix the issue and verify with torch.autograd.gradcheck or by checking gradient flow.
Pro tip: Always enable anomaly detection (torch.autograd.set_detect_anomaly(True)) to pinpoint where the backward pass fails, and remember that .item() detaches the tensor from the graph—use .detach() only when you intend to stop gradients.
Scan the code for .detach(), .item(), .numpy(), in-place operations (e.g., +=, .add_()), and reductions (sum, mean) that might use the wrong dimension.
Create a small input and run forward and backward passes, checking if gradients are None or incorrect. Use torch.autograd.gradcheck to verify.
Remove unnecessary .detach() or .item(), correct the reduction dimension, or replace in-place operations with out-of-place equivalents.
Re-run the backward pass and compare gradients to a reference implementation or use gradcheck. Ensure all parameters receive gradients.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
The clarification step actually saved me here.
First, clarify with the interviewer which positional encoding variant is in use, as the bug differs by type. Then systematically inspect the initialization code for that variant, comparing against known correct implementations to identify the defect. Finally, propose a concrete fix and explain how you would verify it.
Pro tip: Demonstrate deep understanding by noting that initialization bugs often manifest as subtle training instabilities or degraded long-range performance, not immediate crashes. Mention that you would add unit tests for PE properties (e.g., relative distances for RoPE, monotonic decay for ALiBi) to catch such issues early.
Ask the interviewer which positional encoding is used (sinusoidal, RoPE, ALiBi, or learnable) to narrow down the potential bug space.
Examine the code that initializes the positional encoding, looking for common mistakes such as incorrect frequency computation, wrong scaling, or improper caching.
Pinpoint the specific bug by comparing against the correct mathematical formulation or reference implementation, and explain why it's wrong.
Describe the corrected initialization, including any necessary changes to parameters or logic, and ensure it aligns with the intended PE variant.
Outline how to verify the fix, such as unit tests for PE properties, training stability checks, or performance on long sequences.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.