Startups.com·Machine Learning Engineer·Technical Phone Screen·Intermediate

Intermediate

Jun 2026

Summary

Technical screen for an ML Engineer role at Startups.com. One question, pretty deep, basically asked me to reimplement softmax from the ground up in PyTorch with numerical stability and autograd support. Not a warmup question.

Questions Asked (1)

Implement the Softmax function from scratch in PyTorch without using any built-in softmax utilities. Your implementation must handle an arbitrary dim argument, be numerically stable by subtracting the max before exponentiation, and support autograd either through native PyTorch ops or a custom torch.autograd.Function with an explicit backward pass using the softmax Jacobian.

Technical Trade-offsAlgorithms & Data Structures

Author's notes

I started fine.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by clarifying the requirements: arbitrary dim, numerical stability, and autograd support. Then outline a stable forward pass using max subtraction and exp normalization, and discuss two autograd options: using native ops or a custom Function with an explicit backward pass. Finally, walk through the backward derivation and implementation details.

Pro tip: Emphasize that using native PyTorch ops is simpler and less error-prone, but implementing a custom Function demonstrates deeper understanding; mention that the backward pass can be simplified using the identity: grad_input = softmax * (grad_output - sum(grad_output * softmax, dim, keepdim=True)).

1. Clarify requirements and edge cases

Confirm the need for arbitrary dim, numerical stability, and autograd support. Discuss potential edge cases like large inputs or negative values.

2. Implement stable forward pass

Compute max along the specified dim, subtract it for stability, exponentiate, and normalize by the sum of exponentials.

3. Choose autograd approach

Decide between using native PyTorch operations (which automatically support autograd) or writing a custom torch.autograd.Function with an explicit backward pass.

4. Derive and implement backward pass (if custom)

Derive the softmax Jacobian and implement the backward function, using the simplified formula: grad_input = softmax * (grad_output - sum(grad_output * softmax, dim, keepdim=True)).

5. Test and validate

Verify correctness against PyTorch's built-in softmax, check numerical stability with large inputs, and ensure gradients match via torch.autograd.gradcheck.

Key Points to Mention

Numerical stability: subtracting the max prevents overflow in exp.
Arbitrary dim: use dim argument in max, sum, and broadcasting operations.
Autograd: native ops automatically track gradients; custom Function requires explicit backward.
Softmax Jacobian: derivative involves softmax times (delta - softmax) for each pair.
Simplified backward formula: grad_input = softmax * (grad_output - sum(grad_output * softmax, dim, keepdim=True)).
Testing: compare with torch.softmax and use gradcheck for gradient verification.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.