← Openai Interview Insights

Openai·Software Engineer·Technical Phone Screen·Senior

Senior
Apr 2026

Summary

Technical screen for a software engineering role at OpenAI that went deep into neural network math fast. The whole session was basically one long question about backprop from scratch, no libraries allowed, and they wanted working code plus an explanation of the linear algebra.

Questions Asked (1)

Q1

Derive backpropagation for a two-layer network (Affine → ReLU → Affine → softmax), implement the forward pass and analytical gradients in NumPy without autograd, verify them with finite differences, and explain tensor shapes and the chain rule at each step.

Algorithms & Data StructuresTechnical Trade-offsSystem Design
Author's notes

This was the whole interview.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by clearly defining the network architecture, tensor shapes, and forward pass equations. Then derive the gradients step-by-step using the chain rule, implement them in NumPy, and verify with finite differences. Finally, explain the shape transformations and the role of each gradient.

Pro tip: Emphasize the importance of shape consistency and vectorization; use small random inputs for finite difference checks to avoid numerical issues.

1. Define Architecture and Shapes

Specify input dimension D, hidden dimension H, and output dimension C. Write down the shapes of all parameters and intermediate tensors.

2. Forward Pass Implementation

Implement the forward pass in NumPy: affine (XW1 + b1), ReLU, affine (H1W2 + b2), and softmax. Store intermediate values for backward pass.

3. Derive Gradients via Chain Rule

Starting from the loss, compute gradients for each layer: dL/dZ2, dL/dW2, dL/db2, dL/dH1, dL/dZ1, dL/dW1, dL/db1. Show the chain rule steps and shape compatibility.

4. Implement Backward Pass

Code the backward pass in NumPy using the derived formulas, ensuring all matrix multiplications and element-wise operations match the shapes.

5. Verify with Finite Differences

Use numerical gradient checking: perturb each parameter by epsilon, compute loss, and compare the numerical gradient to the analytical one. Use a small tolerance.

Key Points to Mention

  • Tensor shapes at each step: X (N,D), W1 (D,H), b1 (H,), Z1 (N,H), H1 (N,H), W2 (H,C), b2 (C,), Z2 (N,C), probabilities (N,C).
  • Chain rule application: dL/dZ2 = (P - Y)/N for softmax with cross-entropy loss, then backpropagate through affine and ReLU.
  • ReLU derivative: 1 for Z1 > 0, 0 otherwise, applied element-wise.
  • Vectorization: use matrix operations instead of loops for efficiency.
  • Finite difference verification: central difference formula, checking relative error, and handling non-differentiable points of ReLU.
  • Numerical stability: subtract max logits before softmax, and use a small epsilon in gradient checking.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.