Openai·Machine Learning Engineer·Onsite - Coding / Algorithms·Senior

SeniorPrefer not to say

Apr 2026

Summary

ML coding round at OpenAI for a machine learning engineer role. 75 minutes, one big question: implement a Transformer encoder block from scratch in PyTorch without touching any of the built-in attention modules. Brutal but fair.

Questions Asked (1)

Implement a full Transformer encoder block from scratch in PyTorch, including scaled dot-product attention, multi-head attention, a position-wise feed-forward network, residual connections with LayerNorm, and sinusoidal positional encodings. No nn.Transformer or nn.MultiheadAttention allowed. Also discuss attention complexity and how a decoder block would differ.

System DesignAlgorithms & Data StructuresTechnical Trade-offs

Author's notes

This is the whole round, basically.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by outlining the overall architecture and then implement each component in a modular way, testing as you go. Explain the math behind scaled dot-product attention and multi-head attention, and discuss the rationale for design choices like pre-LN vs post-LN. Finally, analyze complexity and contrast with a decoder block.

Pro tip: Mention that you would use torch.einsum for clarity and efficiency in attention computations, and that you would add masking support for decoder self-attention. Also, discuss the trade-offs between pre-LN and post-LN in terms of training stability and performance.

1. Clarify requirements and outline architecture

Confirm that the implementation should be from scratch without nn.Transformer or nn.MultiheadAttention. Sketch the encoder block: multi-head attention, feed-forward network, residual connections, and layer normalization.

2. Implement scaled dot-product attention

Write a function that takes queries, keys, values, and optional mask, computes attention scores, scales by sqrt(d_k), applies softmax, and returns weighted values. Explain the scaling rationale.

3. Implement multi-head attention

Project inputs to multiple heads, apply scaled dot-product attention in parallel, concatenate outputs, and apply a final linear projection. Discuss how this allows the model to attend to different representation subspaces.

4. Build the encoder block

Combine multi-head attention and a position-wise feed-forward network with residual connections and layer normalization. Specify the order (e.g., pre-LN or post-LN) and explain your choice.

5. Add positional encodings and discuss complexity/decoder differences

Implement sinusoidal positional encodings and add them to input embeddings. Analyze the O(n^2) complexity of attention and describe how a decoder block differs (masked self-attention, cross-attention).

Key Points to Mention

Scaled dot-product attention formula: Attention(Q,K,V) = softmax(QK^T / sqrt(d_k)) V, and why scaling is necessary.
Multi-head attention: splitting into heads, parallel computation, concatenation, and output projection.
Residual connections and layer normalization: pre-LN vs post-LN, and their impact on training stability.
Position-wise feed-forward network: two linear layers with ReLU activation in between, applied independently to each position.
Sinusoidal positional encodings: formula using sine and cosine functions of different frequencies, and why they help with sequence order.
Attention complexity: O(n^2 * d) time and O(n^2) memory for sequence length n, and how decoder block differs with masked self-attention and cross-attention.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.