Openai·Software Engineer·Technical Phone Screen·Senior

SeniorPrefer not to say

May 2026Remote

Summary

Interviewed for a SWE role at OpenAI and got a pretty deep ML systems question right out of the gate. The whole thing was focused on implementing multi-head attention from scratch, shapes and all. Not a vibe check, they wanted real code and real reasoning.

Questions Asked (1)

Implement the forward pass of Multi-Head Attention using NumPy or PyTorch. You need to handle splitting into heads, scaled dot-product attention with an optional mask, concatenating outputs, and explaining how you keep the softmax numerically stable. Clearly state tensor shapes throughout.

System DesignTechnical Trade-offsAlgorithms & Data Structures

Author's notes

This was the whole interview basically.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by outlining the overall computation flow and tensor shapes, then implement each component (linear projections, head splitting, scaled dot-product attention with optional mask, softmax stabilization, concatenation, and output projection) in a clear, modular way. Emphasize numerical stability by subtracting the row-wise maximum before exponentiation and using a large negative value for masked positions.

Pro tip: Mention that you would use `torch.nn.functional.scaled_dot_product_attention` in production for efficiency, but implement manually to demonstrate understanding. Also, note that masking should be applied before softmax to avoid NaNs.

1. Define input shapes and linear projections

Clarify input tensor shapes (batch_size, seq_len, d_model) and project to queries, keys, values using learned weight matrices to get (batch_size, seq_len, d_model) for each. Mention that d_model = num_heads * d_k.

2. Split into multiple heads

Reshape Q, K, V from (batch_size, seq_len, d_model) to (batch_size, num_heads, seq_len, d_k) by viewing and transposing. Explain that this allows parallel attention computations per head.

3. Compute scaled dot-product attention with optional mask

Compute attention scores as Q @ K^T / sqrt(d_k), shape (batch_size, num_heads, seq_len, seq_len). Apply mask (if provided) by setting masked positions to a large negative value (e.g., -1e9) before softmax. Then compute softmax along the last dimension with numerical stability by subtracting the max per row.

4. Apply attention weights to values and concatenate heads

Multiply attention weights (after softmax) with V to get (batch_size, num_heads, seq_len, d_k). Transpose and reshape back to (batch_size, seq_len, d_model) to concatenate heads.

5. Final linear projection and output

Apply a final linear layer to the concatenated output to produce the final result of shape (batch_size, seq_len, d_model). Optionally mention dropout.

Key Points to Mention

Tensor shapes at each step: input (B, L, D), after projection (B, L, D), after head split (B, H, L, d_k), attention scores (B, H, L, L), output (B, L, D).
Scaling factor 1/sqrt(d_k) to prevent softmax saturation.
Numerical stability in softmax: subtract max before exponentiation.
Masking: use large negative value (e.g., -1e9) before softmax, not after.
Efficiency: use batched matrix multiplication and avoid loops.
Optional: mention that in PyTorch, one can use `torch.nn.functional.scaled_dot_product_attention` for optimized implementation.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.