OpenAI·Machine Learning Engineer·Technical Phone Screen
May 2026
OpenAI ML Engineer round focused entirely on gradient-based optimization and PyTorch autograd. Pretty deep technically, more math-on-the-board than I expected for a coding round.
- Derive backpropagation by hand for a small network with one or two hidden layers using sigmoid or ReLU activations, a softmax output, and cross-entropy loss. Walk through the forward pass, then apply the chain rule backward to compute the weight and bias gradients at each layer.
- Implement the forward pass and a manual backward pass in NumPy for a small neural network, then verify your computed gradients match what PyTorch autograd produces.
- Explain the correct usage of PyTorch autograd: when to call backward(), what happens to gradients across training iterations without zero_grad(), and what detach() and torch.no_grad() are actually doing.
- Identify and explain common PyTorch autograd bugs: forgetting to zero gradients, performing in-place operations on leaf tensors, and trying to call backward on a computation graph that has already been freed.
- How does gradient checking via finite differences work, and when would you use it?
- Why do vanishing and exploding gradients happen, and how do weight initialization strategies like Xavier or He initialization help address them?
- What problems do BatchNorm and LayerNorm solve during training, and how do they interact with the gradient flow?
“This took way longer than I wanted.”