Microsoft·Software Engineer·Technical Phone Screen·Senior

Senior

May 2026

Summary

Two-part technical deep-dive for an AI Infrastructure role at Microsoft, both questions were heavy on training internals. The kind of interview where you realize halfway through that you should have reviewed your IEEE-754 notes more recently.

Questions Asked (2)

Derive how gradients flow through a residual connection y = x + F(x). Show that ∂L/∂x = ∂L/∂y · (1 + ∂F/∂x), explain why the identity shortcut helps with vanishing gradients, and discuss how pre-norm vs post-norm LayerNorm placement changes things.

System DesignTechnical Trade-offs

Author's notes

The derivation itself wasn't too bad once I wrote it out.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by deriving the gradient using the chain rule, emphasizing the additive identity term. Then explain how this additive term mitigates vanishing gradients by providing a direct path for gradient flow. Finally, compare pre-norm and post-norm LayerNorm placements, discussing their impact on gradient propagation and training stability.

Pro tip: Connect the mathematical derivation to practical implications: mention that pre-norm often allows training deeper networks without warmup, while post-norm may require careful initialization and learning rate tuning.

1. Derive the gradient

Apply the chain rule to y = x + F(x) to get ∂L/∂x = ∂L/∂y · (1 + ∂F/∂x). Clearly state the assumption that F is differentiable.

2. Explain vanishing gradient mitigation

Highlight that the '1' term ensures a direct gradient path even if ∂F/∂x is small, preventing gradients from vanishing in deep networks.

3. Discuss pre-norm vs post-norm

Define pre-norm (LayerNorm before sublayer) and post-norm (after residual addition). Explain how pre-norm keeps the identity path clean, while post-norm can attenuate gradients due to normalization on the residual sum.

4. Connect to training dynamics

Mention that pre-norm often enables training deeper models without learning rate warmup, whereas post-norm may require warmup and careful initialization.

Key Points to Mention

Chain rule application and the additive gradient term
The role of the identity shortcut in providing a gradient highway
Vanishing gradients in deep networks and how residual connections alleviate it
Pre-norm vs post-norm LayerNorm placement and their effect on gradient flow
Practical implications: training stability, warmup, and depth scalability
Potential trade-offs: pre-norm may reduce expressiveness, post-norm can be harder to train

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.

Walk through converting an FP32 model to FP16 or BF16: cover the IEEE-754 layout differences in terms of range vs precision, where overflow and underflow tend to occur (softmax, gradient accumulation, loss), why mixed-precision training keeps a master FP32 weight copy, how dynamic loss scaling works, and how BF16 changes the trade-offs relative to FP16. Be prepared to compute representable ranges and identify which operations you'd keep in FP32.

Technical Trade-offsSystem Design

Author's notes

This one is where I felt the pressure.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by comparing the IEEE-754 layouts of FP32, FP16, and BF16 to highlight the range vs precision trade-off, then walk through where overflow/underflow occur in practice (softmax, gradient accumulation, loss), and finally explain mixed-precision training with master weights and dynamic loss scaling, emphasizing how BF16 simplifies the process. Be ready to compute representable ranges and specify which operations stay in FP32.

Pro tip: Mention that BF16 has the same exponent range as FP32, so it rarely needs loss scaling, but its lower precision can hurt convergence for some models—this shows you understand the nuanced trade-offs beyond just 'BF16 is better'.

1. Compare IEEE-754 layouts

Explain the bit allocation: FP32 (1 sign, 8 exponent, 23 mantissa), FP16 (1, 5, 10), BF16 (1, 8, 7). Emphasize that FP16 sacrifices range for precision, while BF16 sacrifices precision for range.

2. Compute representable ranges

Calculate approximate ranges: FP16 max ~65504, min normal ~6.1e-5; BF16 max ~3.4e38, min normal ~1.2e-38. Note that FP16 underflows to zero below ~6e-8 (subnormals) and overflows to inf above 65504.

3. Identify overflow/underflow hotspots

Discuss operations prone to issues: softmax (large exponentials overflow FP16), gradient accumulation (small gradients underflow), and loss computation (large losses overflow). Mention that these often require FP32.

4. Explain mixed-precision training

Describe keeping a master FP32 copy of weights for accurate updates, casting to FP16/BF16 for forward/backward, and using FP32 for reductions and sensitive ops. Highlight that master weights prevent precision loss in updates.

5. Detail dynamic loss scaling and BF16 trade-offs

Explain dynamic loss scaling: multiply loss by a scale factor to shift gradients into FP16 range, check for inf/NaN, and adjust scale. Then contrast BF16: no loss scaling needed due to FP32-like range, but lower precision may require other mitigations.

Key Points to Mention

IEEE-754 bit layouts: FP32 (8 exponent, 23 mantissa), FP16 (5 exponent, 10 mantissa), BF16 (8 exponent, 7 mantissa).
FP16 range: ~6.1e-5 to 65504; BF16 range: ~1.2e-38 to 3.4e38 (same as FP32).
Overflow/underflow hotspots: softmax (exp overflow), gradient accumulation (underflow), loss (overflow).
Mixed-precision training: master FP32 weights, FP16/BF16 compute, FP32 for reductions and sensitive ops.
Dynamic loss scaling: scale loss to prevent underflow, skip step on inf/NaN, adjust scale dynamically.
BF16 trade-offs: no loss scaling needed, but lower precision may affect convergence; often preferred on hardware that supports it.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.