NVIDIA·Software Engineer·Technical Phone Screen·Senior

Senior

Apr 2026

Summary

Technical phone screen at NVIDIA for a software engineering role, pretty deep into ML fundamentals and GPU compute. The question was one long multi-part thing about MLP forward passes and PyTorch internals, which felt more like a grad school exam than a typical SWE interview.

Questions Asked (1)

Walk through a full MLP forward pass: write out the linear transform Y = XW + b with explicit shapes for W and b, explain how PyTorch broadcasting handles a bias vector of shape (d_h,) across a batch, chain through multiple layers with a nonlinear activation like ReLU while tracking shapes at each step, and explain how these operations vectorize on GPU. Also discuss the differences between torch.matmul, the @ operator, and nn.Linear, and common pitfalls with shapes like (B, 1) vs (B,).

System DesignTechnical Trade-offsAlgorithms & Data Structures

Author's notes

This was a single question but it just kept going.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by defining the MLP forward pass with explicit shapes, then walk through a concrete example with a batch, showing how broadcasting applies the bias and how ReLU is applied elementwise. Next, explain how PyTorch vectorizes these operations on GPU using batched matrix multiplication and elementwise kernels. Finally, compare torch.matmul, @, and nn.Linear, and highlight common shape pitfalls like (B,1) vs (B,).

Pro tip: Emphasize that nn.Linear is a high-level module that encapsulates weight initialization, device placement, and optimized kernels, while torch.matmul and @ are lower-level ops; knowing when to use each shows depth. Also, mention that avoiding shape (B,1) vs (B,) bugs often involves using .squeeze() or .unsqueeze() appropriately, and that broadcasting can silently produce wrong shapes if not careful.

1. Define the linear transform with shapes

Write Y = XW + b, specify X shape (B, d_in), W shape (d_in, d_h), b shape (d_h,), and Y shape (B, d_h). Explain that b is broadcast across the batch dimension.

2. Explain broadcasting and activation

Describe how PyTorch broadcasting adds b to each row of XW, and then apply ReLU elementwise: A = max(0, Y). Track shapes through multiple layers, e.g., (B, d_in) -> (B, d_h) -> (B, d_out).

3. Discuss GPU vectorization

Explain that matrix multiplication is batched and parallelized on GPU using cuBLAS, and elementwise ops like ReLU are fused kernels that operate on all elements concurrently.

4. Compare matmul, @, and nn.Linear

Clarify that torch.matmul and @ are equivalent for 2D tensors, but @ is syntactic sugar; nn.Linear is a module that holds weights and bias, handles initialization, and calls the underlying matmul with optimized kernels.

5. Highlight shape pitfalls

Discuss common issues: (B,1) vs (B,) can cause unintended broadcasting or errors; use .squeeze() or .unsqueeze() to align dimensions, and be mindful of batch dimensions in higher-rank tensors.

Key Points to Mention

Explicit shapes: X (B, d_in), W (d_in, d_h), b (d_h,), Y (B, d_h)
Broadcasting: b is added to each row of XW, effectively replicating b across the batch
ReLU activation: elementwise max(0, x), preserving shape
GPU vectorization: batched matmul via cuBLAS, elementwise ops as parallel kernels
torch.matmul vs @: functionally equivalent for 2D, @ is operator overload
nn.Linear: encapsulates weights, bias, initialization, and optimized forward pass
Shape pitfalls: (B,1) vs (B,) can lead to silent broadcasting errors; use squeeze/unsqueeze

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.