Startups.com Interview Questions & Experiences

Startups.com·Machine Learning Engineer·Technical Phone Screen

Jul 2026

Technical screen for an ML Engineer role at Startups.Com, heavy on PyTorch internals and distributed training. Four questions, all deep end of the pool, no warmup.

You have a PyTorch training script for CIFAR-10 that either fails to converge, produces NaN/Inf loss, or trains way slower than it should. Walk through how you'd systematically debug it.
How would you scale that same training job to multiple GPUs using Fully Sharded Data Parallel (FSDP)? Cover initialization, model wrapping, optimizer state, gradient accumulation, and checkpointing.
How would you implement or approximate sparse gradient all-reduce across workers? What communication patterns would you use, what are the tradeoffs versus dense all-reduce, and when does it actually make sense to do this?
Under what circumstances would you write a custom CUDA or Triton kernel to speed up training, and how would you verify both the performance gain and correctness?

“This one I actually felt decent about.”

View Post

Startups.com·Machine Learning Engineer·Technical Phone Screen

Jun 2026

Technical screen for an ML Engineer role at Startups.com. One question, pretty deep, basically asked me to reimplement softmax from the ground up in PyTorch with numerical stability and autograd support. Not a warmup question.

Implement the Softmax function from scratch in PyTorch without using any built-in softmax utilities. Your implementation must handle an arbitrary dim argument, be numerically stable by subtracting the max before exponentiation, and support autograd either through native PyTorch ops or a custom torch.autograd.Function with an explicit backward pass using the softmax Jacobian.

“I started fine.”

View Post

Startups.com·Machine Learning Engineer·Technical Phone Screen

Jun 2026

Technical screen for an ML Engineer role, focused entirely on Transformer attention mechanisms. Pretty deep dive, they clearly wanted someone who understood not just the math but the practical tradeoffs for inference.

Walk me through scaled dot-product attention mathematically, and explain why the scaling factor is there. What are the tensor shapes for Q, K, V in a batched setting?
How does multi-head attention differ from single-head attention? Cover the projection matrices, per-head computation, concatenation, and the output projection. Also walk through compute and memory complexity in terms of sequence length, number of heads, and head dimension.
What is grouped-query attention and how does it sit between standard multi-head attention and multi-query attention?
In a real deployment scenario, when would you actually choose MHA over GQA or MQA? Think about model quality, KV cache size, memory bandwidth, and any practical constraints.

“The math itself was fine, softmax over QK^T divided by sqrt(d_k) times V.”

View Post

Startups.com·Machine Learning Engineer·Onsite - System Design / Architecture

Jun 2026

System design round at Startups.com for an ML Engineer role, focused entirely on building an AI-powered database performance advisor. One long, dense question that kept branching into sub-problems every time I thought I'd covered it.

Design a system that accepts natural-language questions about database performance, translates them to SQL, executes the queries safely against the database or its metadata, analyzes the results, and surfaces optimization recommendations to the user. Walk through the NL-to-SQL layer, execution controls, result analysis, recommendation generation, evaluation, and observability.

“This felt like four system design questions stapled together.”

View Post

Startups.com·Machine Learning Engineer·Technical Phone Screen

May 2026

ML engineer interview at Startups.Com that went deep into transformer inference internals, specifically KV caching. Pretty much the whole session was one long technical design question with several sub-parts. Not a lot of small talk, they clearly wanted to see if you actually understand what's happening under the hood.

What is the KV cache in a decoder-only Transformer, what tensors get stored per layer, and how does it change the computation during incremental decoding?
Walk through an implementation plan for KV caching that handles variable-length sequences in a batch, supports beam search or speculative decoding where sequences can branch, and scales to very long contexts like 32k to 128k tokens.
What are the key performance considerations when implementing a KV cache, covering memory layout, avoiding copies and reallocations, interaction with fused attention kernels, and precision choices for the cached tensors?
What are common correctness bugs when adding a KV cache to a transformer, such as issues with attention masking, positional encodings, shape mismatches, or other subtle errors?

“This part felt okay.”

View Post

Startups.com·Machine Learning Engineer·Technical Phone Screen

May 2026

Technical screen for an ML Engineer role at Startups.Com, focused entirely on implementing neural network primitives from scratch in PyTorch without touching torch.nn. Pretty deep dive, felt more like a take-home style problem compressed into a live setting.

Implement ReLU from scratch using only basic PyTorch tensor operations, without using any torch.nn or torch.nn.functional calls.
Implement numerically stable softmax from scratch for an arbitrary dimension.
Implement LayerNorm from scratch, including learnable gamma and beta parameters, normalizing over the last k dimensions.
Implement BatchNorm for both training and inference modes from scratch, including running mean/variance updates with momentum.
Implement RMSNorm from scratch. How does it differ from LayerNorm and when might you prefer it?
For each of the components you implemented, how would you verify correctness? Walk through your testing strategy including numerical checks and edge cases.

“Easiest one on the list.”

View Post

Startups.com·Machine Learning Engineer·Technical Phone Screen

Apr 2026

Interviewed for an ML Engineer role at Startups.com and got a pretty deep PyTorch implementation question. Not a vibe check, they wanted actual code and a real understanding of autograd internals.

Implement the ReLU activation function from scratch in PyTorch without using built-in ReLU utilities. Your solution should handle tensors of any shape, apply elementwise max(0, x), and support autograd with gradients of 1 where x > 0 and 0 elsewhere. Show both a plain tensor function version and a custom torch.autograd.Function with explicit forward and backward methods.

“I knew ReLU conceptually but writing the autograd.Function version from memory was rougher than expected.”

View Post