ReflectionAI·Software Engineer·Technical Phone Screen·Senior

Senior

Jun 2026

Summary

Research Engineer interview at ReflectionAI covering LLM fundamentals pretty deeply, transformer internals, RL-based post-training methods, and then a specific technical drill-down on parallelism strategies for inference.

Questions Asked (1)

If you have a single decoding request to serve (no batching), which parallelism strategy would you pick among data parallelism, tensor parallelism, and pipeline parallelism, and why? Walk through the tradeoffs in latency, memory, and underutilization.

System DesignTechnical Trade-offs

Author's notes

This is the kind of question that sounds easy until you start talking and realize you've been thinking about parallelism mostly in the training context.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by clarifying that with a single request and no batching, latency is the primary constraint, so you should favor strategies that minimize communication overhead and maximize per-GPU utilization. Then compare data, tensor, and pipeline parallelism on latency, memory, and underutilization, ultimately recommending tensor parallelism for intra-layer scaling or data parallelism if the model fits on one GPU.

Pro tip: Emphasize that pipeline parallelism introduces bubbles and sequential dependencies that are disastrous for single-request latency, while tensor parallelism adds all-reduce overhead per layer—so the best choice depends on whether the model fits in a single GPU's memory.

1. Clarify the scenario and constraints

Restate that there is a single decoding request, no batching, and the goal is to minimize latency while respecting memory limits. Identify whether the model fits on one GPU, as this determines the viable options.

2. Evaluate data parallelism

Explain that data parallelism replicates the model across GPUs and splits the batch, but with a single request there is no batch to split, so it offers no benefit and wastes memory. It is only useful if you need to serve multiple concurrent requests.

3. Evaluate tensor parallelism

Describe how tensor parallelism splits individual layers across GPUs, reducing per-GPU memory and enabling larger models. However, it introduces communication overhead (e.g., all-reduce) per layer, which can increase latency, especially with slow interconnects.

4. Evaluate pipeline parallelism

Explain that pipeline parallelism splits the model into stages across GPUs, but for a single request it creates pipeline bubbles and sequential dependencies, leading to high latency and GPU underutilization. It is better suited for throughput with many micro-batches.

5. Synthesize and recommend

Conclude that for a single request, if the model fits on one GPU, use no parallelism (or data parallelism with a single replica). If it doesn't fit, tensor parallelism is preferred over pipeline parallelism because it avoids bubbles and can be optimized with high-bandwidth interconnects.

Key Points to Mention

Latency is the dominant metric for single-request decoding; throughput is irrelevant.
Data parallelism provides no benefit for a single request because there is no batch to split.
Tensor parallelism reduces per-GPU memory but adds communication overhead per layer, which can increase latency.
Pipeline parallelism introduces pipeline bubbles and sequential execution, causing high latency and GPU underutilization for a single request.
The choice depends on whether the model fits in a single GPU's memory; if it does, avoid parallelism entirely.
For models that don't fit, tensor parallelism with high-bandwidth interconnects (e.g., NVLink) is the best option among the three.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.