Applied intuition·Machine Learning Engineer·Technical Phone Screen·Senior

Senior

Jun 2026

Summary

Applied Intuition ML engineer interview with a deep technical question on Transformer attention masking. Single question but it went pretty far into implementation details, edge cases, and debugging, more thorough than I expected for what felt like a phone screen.

Questions Asked (1)

Implement attention masking for an autoregressive Transformer that handles variable-length sequences with padding. Build both a causal mask and a padding mask, show how to combine them, and explain what bugs look like in practice and how you'd test for them.

System DesignTechnical Trade-offsAlgorithms & Data Structures

Author's notes

This one went deeper than I was ready for.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by clarifying the requirements: autoregressive decoding, variable-length sequences with padding, and the need for both causal and padding masks. Then describe how to construct each mask, combine them via logical OR (or addition with -inf), and apply them to attention scores. Finally, discuss common bugs, their symptoms, and a testing strategy that includes unit tests and gradient checks.

Pro tip: Emphasize that padding masks must be applied to both keys and queries to avoid attending to padding tokens, and that using -inf before softmax is safer than multiplying by 0 to prevent NaNs. Also, mention that mask shapes must broadcast correctly across batch and head dimensions.

1. Clarify requirements and assumptions

Confirm that the model is autoregressive (decoder-only), sequences are padded to the same length within a batch, and padding tokens should be ignored in attention. Ask about mask dtype (boolean vs. additive) and whether the implementation uses PyTorch, TensorFlow, or custom code.

2. Construct causal and padding masks

For causal mask, create a lower-triangular matrix of shape (seq_len, seq_len) with True/1 for allowed positions. For padding mask, create a mask of shape (batch_size, seq_len) where True/1 indicates valid tokens (or False/0 for padding), then expand to (batch_size, 1, 1, seq_len) for broadcasting.

3. Combine masks and apply to attention scores

Combine causal and padding masks using logical OR (if boolean) or addition (if additive with -inf). Apply the combined mask to attention scores before softmax: scores = scores.masked_fill(mask == 0, -inf) or scores + mask. Ensure the mask broadcasts correctly across batch and head dimensions.

4. Explain common bugs and symptoms

Describe bugs like: attending to future tokens (causal mask not applied), attending to padding tokens (padding mask missing or misaligned), NaNs from softmax on all -inf rows, and incorrect broadcasting leading to shape errors. Symptoms include degraded generation quality, loss not decreasing, or NaN losses.

5. Outline testing strategy

Propose unit tests: check mask shapes, verify that masked positions have zero attention weight after softmax, test with variable-length sequences and padding, and use gradient checks to ensure no gradients flow to padding tokens. Also, test edge cases like sequence length 1 and all-padding sequences.

Key Points to Mention

Causal mask ensures each position attends only to previous positions; typically a lower-triangular boolean matrix.
Padding mask prevents attention to padding tokens; must be applied to keys and queries, and broadcast across batch and head dimensions.
Combine masks via logical OR (boolean) or addition with -inf (additive); apply before softmax.
Use -inf instead of 0 multiplication to avoid NaNs when a row is fully masked (e.g., all padding).
Common bugs: missing causal mask (future leakage), missing padding mask (attending to pad), incorrect broadcasting, and NaNs from softmax on all -inf rows.
Testing: unit tests for mask shapes and attention weights, gradient checks for padding tokens, and integration tests with variable-length sequences.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.