Openai·Machine Learning Engineer·Technical Phone Screen·Senior

Senior

Apr 2026

Summary

ML coding round at OpenAI for an MLE role. The whole thing was a debugging exercise on a broken Transformer implementation, which sounds manageable until you're staring at someone else's messy code trying to remember whether pre-LN or post-LN even matters.

Questions Asked (1)

You're given a buggy Transformer implementation. Find all the bugs, explain why each one is wrong, and fix them. Issues may include attention masking, scaling, residual connections, tensor shapes, dropout, or positional encoding.

Root Cause AnalysisTechnical Trade-offsAlgorithms & Data Structures

Author's notes

This is genuinely hard under pressure.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by outlining the Transformer architecture and the typical bug-prone areas, then systematically inspect each component (attention, residual, dropout, positional encoding) for correctness. For each bug, explain the expected behavior, why the current implementation is wrong, and provide a corrected version with justification.

Pro tip: Demonstrate deep understanding by not only fixing bugs but also explaining how each bug would manifest during training (e.g., loss divergence, poor performance) and how to test for it.

1. Review the overall architecture

Sketch the Transformer's data flow and identify all components that could contain bugs, such as attention, feed-forward, residual connections, layer normalization, dropout, and positional encoding.

2. Inspect attention mechanism

Check scaling of QK^T, masking (padding and causal), softmax dimension, and tensor shapes for multi-head attention. Verify that the scaling factor is 1/sqrt(d_k) and masks are applied correctly before softmax.

3. Examine residual connections and layer normalization

Ensure residual connections are added correctly (e.g., x + sublayer(x)) and layer normalization is applied appropriately (pre-norm vs post-norm). Check for missing residual connections or incorrect order.

4. Check dropout and positional encoding

Verify dropout is applied only during training and in the correct places (after attention, after feed-forward, on embeddings). Ensure positional encoding is added to embeddings and not mixed up with other tensors.

5. Validate tensor shapes and fix bugs

Trace tensor shapes through the network to catch shape mismatches. For each bug found, explain the correct implementation and provide a fix, ensuring consistency with the original Transformer paper.

Key Points to Mention

Attention scaling: divide QK^T by sqrt(d_k) to prevent softmax saturation.
Masking: apply padding mask (for padded tokens) and causal mask (for decoder) before softmax, using -inf for masked positions.
Residual connections: ensure they are added around each sublayer (attention and feed-forward) and that layer normalization is applied correctly (pre-norm or post-norm).
Dropout: apply dropout to sublayer outputs and embeddings, but only during training.
Positional encoding: add to input embeddings, typically using sine and cosine functions of different frequencies.
Tensor shapes: verify dimensions for multi-head attention (batch, heads, seq_len, d_k) and ensure linear layers project to correct sizes.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.